In order to see and to set the starting number run the following SQL

SELECT * FROM staglog WHERE stgrfnbr=0

This is the base record you will always have one in staglog and you should never delete it.

Update the entry in the stgpkey field in order to change the starting record where staglog will start looking for entries where stgprocessed = 0.

Although this was not the first issue I encountered but the last, the first thing to do is make sure the JDK you use is 1.6.  I initially had 1.7 setup it was the latest and greatest when I followed the instructions for setting up the SDK that the Info Centre pointed me at.  Given there are no instructions on how they have written the sample app on the IBM site, made sense to me.  But once I resolved the two problems below I had a whole load of red errors about

the method onCLick(view) of type new …

A Google on this indicated along the lines of this, especially when I looked at the code flagging the errors

Since Java 1.6 the annotation @Override is not only possible for inherited methods from a superclass but also for implemented methods from an interface. But if your compiler is still set to 1.5 the problem is that he will check if you’re overwriting a method from a super class.

So my guess is in 1.7 something changed again.  Once I got the 1.6 JDK installed and set up within Eclipse and cleaned the project all the errors went, just a few warnings left.  So get 1.6 for your JDK not 1.7.

Next when I installed the Android SDK it download the latest API 14 files. I installed Eclipse set the Path and as soon as I imported the Android Application I got the following error

Madisons-AndroidNativeMobile Unable to resolve target ‘Google Inc.:Google APIs:8′

What this means is I am missing the Level 8 API’s and have to import them into the Eclipse environment. You can either use the Window – Android SDK Manager option within the Eclipse environment. Otherwise use the Android SDK Manager itself.  I did find that until I closed and then opened the Eclipse environment I still had the error, not sure if you will get the same thing.  Just make sure the SDK manager reloads everything OK.

Then when I restarted I now saw a different error, now this is due to me having 1.7 installed but worth a mention.  if you go with 1.6 you won’t see this.

Madisons-AndroidNativeMobile] Android requires compiler compliance level 5.0 or 6.0. Found ’1.4′ instead. Please use Android Tools > Fix Project properties.

You can find this by right clicking on the project and it is near the bottom if the options.  if you look in the properties of the application as well and click on ‘Java Compiler’  you can see the compiler compliance level if you let the environment do the setting it goes to 1.5 (V5) you can select 1.6 (6) this seems to be OK.  Not sure if it is best to be on 6 or not?

Hopefully I can now build and deploy the app and see what it does.

I have the Madisons sample store deployed on a server and I was using Firefox to get into Management Center. However when I wanted to create file assets I got an error saying it could not complete.

Looking at the systemout.log file I could see a

wcfServiceDataset onIOError while invoking URL [/lobtools/cmc/AttachmentFileUpload

Nothing obvious with a Google search on that error. I then spotted this link in the Info Center.  It would have exactly the problem because this is a self signed certificate for the Madisons store that I am using on the server.  It would be nice if the error was handled better, I swapped to Internet Explorer and the file upload works fine in there.

CMN1245E: Server fails to redirect to the view URL

And a generic error being displayed in the store page.  A google on the CMN1245E error was showing nothing however after opening a PMR a link to the following was provided.  hopefully this can be expanded to cover CMN1245E as well.

Receive CMN0207E error indicating incorrect URL when redirect occurs in WebSphere Commerce

The same fix applied in this case the webserver is remote to the application server, so the hostname is different that is being used.  Once the extra paramater was specified from the Enable URL Filtering section in the Info Centre into the instance.xml and the server restarted the site worked fine, all links were now clickable.

However if you are doing this with a remote webserver providing access to the ApplicationServer, you will find that it uses the HOSTNAME of the application server where you run the command as the hostname for for later commands.  It also uses it for the configuration on how the pages build within the site, when they want to access the search engine.  This means that when you run the buildindex command or the Madisons sample store things will fail, and you will get a remote host error being displayed such as the following when running the buildindex command.

INFO: I/O exception (java.net.ConnectException) caught when processing request:
A remote host refused an attempted connect operation.
04-Jul-2011 21:35:33 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: A remote host refused an attempted connect operation.

It would be nice if on startup the server configuration showed the hostname that would be used for SOLR, it would provide a much quicker clue.

What you need to do is make sure you update the following tables SRCHCONF and SRCHCONFEXT and replace the host name with that of your webserver.  Then update the wc-search.xml file found in /xml/config/com.ibm.commerce.catalog-fep in the deployed application, make sure you redeploy the file after changes.  You will want to restart the Application servers to pick these details up both your SOLR and Commerce server.

You should then find that your search works fine and you can setup the index.  It does need IBM to add an option onto the setupSearchIndex command for hostname, that would remove these steps.  Remember if you run the setup command again it will go back to how it was in the DB and the XML file, so you need to make the same changes.

I wanted to setup a development environment with SOLR and although a lot of this info is taken from the V7 info Centre, it left me puzzling about the actual steps to perform.  This was from the point of view of installing Feature Pack 2 and what I then needed to do to see SOLR in action.

1.) I started with a clean install of the WC Toolkit for V7 that was installed into the latest RAD version (7.5.5.3 iFix001), using the IBM installation manger.

2.) I applied Fixpack 7.0.0.2 onto this environment, again using the Installation manager.

3.) I then applied Feature Pack 2 for V7 and again used the Installation manager.  this gave me the option of which features to install and I selected “All” of them to be applied to the environment.  This was the step that was now a little confusing because I knew from the server installation that I had processes to do with SOLR, after enabling the foundation feature. But when you read the IC it is not clear what point you are now at with the environment.

4.) I looked at the following link, for Deploying WebSphere Commerce Search Locally.  This page had been linked to from enabling the foundation feature, so did the fact that I had enabled the feature mean it was complete?  The answer is yes you have no need to do this step if you enabled features from IBM Installation manager tool.  I started up my commerce server within the development environment and was able to run the request “http://localhost/solr/Default/select?q=*%3A*” and get the correct response.

5.) Now I needed to run the setupSearchIndex.bat -masterCatalogId 10001 command for the Madisons FEAP 2 store I have created.  What I did find is that I now must stop the commerce server in the development environment otherwise the Apache database is locked and it will not run.

6.) Now run the di-preprocess.bat file against the search index, again I kept the server stopped at this point.

7.) Finally I ran the di-buildindex.bat -masterCatalogId 10001 this command needs to talk to the SOLR environment, so you must have your server started when it runs.  It will also take a while to run depending on the data you have compared to the other two commands.

8.) Once that is there you should now be able to access your development environment store and see it displayed with the SOLR.  One thing I have found though is that if you have SOLR enabled and even with the Apache database I have started to see Out of Memory errors occurring in the environment.

There was still a lot of learning and some quite major changes to make, that you don’t really don’t come across when you use Dynacache within the WebSphere Application Server server for a Commerce environment.

If you are wondering what is Dynacache and you use WebSphere Commerce then it is important that you understand, not running Cache in a Commerce environment is a little like having a fast car and only using your first three gears maybe.  What is possible will always be limited and will not do the environment any good at all, affecting the end users of the site.  You will never really maximise the performance that is possible from your Commerce environment, have a look at this Redbook to get an understanding ‘Mastering Dynacache in WebSphere Commerce‘.  I have also split this blog post into two parts, just so we do not end up with a huge page to read and get bored in, perhaps it could even be three.

The aim of WebSphere Commerce and WebSphere Extreme Scale

This article is not about the details of WXS there are some excellent pieces available such as ‘why WXS‘ for that, especially from Billy Newport. There is also a Elastic Cache redbook which gives some excellent background and also some WC integration options plus part of its team was Jonathan Marshall, who helped us along the WXS path.

For the environment we decided to go for WXS 7.1 and have kept that up to date as well with the latest fixpacks so 7.1.0.2 is the current version.  We then went for the standalone process model rather than using WebSphere to provide the WXS grid, that model is discussed in the redbook.  There are advantages to running on WAS, but it seemed to make sense plus save on licences just using the standalone process mode for the WXS grid on Ubuntu (see previous blog for setup).

The aim of integrating WXS with Commerce was to improve several areas.

• Capability to hold more cache items as the products available was going to grow substantially on the stores.
• Reduce the overhead on the application servers when dealing with the cache, such as disk offload and the space the cache takes up within the JVM.
• Improve startup times for servers when coming on line as the cache would already exist rather than having to build.

For the grid setup we initially have three machines each running a WXS catalogue service process and then having three containers processes on each machines.  This is a starting point and will be further tuned, but it provides the ability to balance in terms of resources and failure.

WebSphere Application Server Configuration

On the application server we had installed WXS onto each machine, and created a catalogue service domain with the three catalogue service processes defined.  It’s worth noting here that the name you give the grid means nothing, you can call it anything you would like.

Also we deployed the enhanced cache monitor rather than the standard version, as this provides a better view of the cache environment when using WXS.

Underneath each of the servers in the environment the following properties need to be added to the Java Virtual Machine custom settings.  They are defining the change in cache environment that will be used and its setup, also the cache is remote.  The first part in the following is the name of the property and after the / is the value.  There may be other properties in there already, depending on your setup.  One setting that was interesting is that you need to tell WXS to turn compression on.

com.ibm.websphere.xs.dynacache.disable_recursive_invalidate  /   true
com.ibm.websphere.xs.dynacache.enable_compression   /    true
com.ibm.websphere.xs.dynacache.ignore_value_in_change_event   /  true
com.ibm.websphere.xs.dynacache.num_initial_containers /  1
com.ibm.websphere.xs.dynacache.topology    /     remote
com.ibm.ws.cache.CacheConfig.cacheProviderName   /  com.ibm.ws.objectgrid.dynacache.CacheProviderImpl
com.ibm.ws.cache.CacheConfig.enableCacheReplication   /  true

One last thing is that when you go into the console even with the fixes applied it will still show the WXS level shown as 7.1.0.0.  So you need to look at the systemout.log on startup to really see what fix level has been applied.

Dynacache Configuration

The dynacache configuration was the point at which, we found that the most serious changes were going to have to be made yet we had assumed it would be minimal.  Problems you will have are not apparent when you run cache within the JVM of the application server, compared to how it will look with a remote grid.  When you use WXS it is the case lots of bad requests or constant chatter for cache content is going to add overhead when the grid is remote.  Even if the grid sits on Gigabit ethernet then ‘remote’ is remote and there will be network overhead.  What we saw initially is that with local dynacache pages would take a second or so to build transfer that to a remote cache page and suddenly it was twelve seconds or more.  Yet the only change was the introduction of WXS, so what could be going on?  What then followed was a lot of time with the support team and here is what we found out.

WebSphere Commerce Data Caches

First we had to implement WebSphere Commerce Data Caches, it is a good article but to me not the most obvious to follow on how you do this.  What it is simply saying is that there are certain types of data you can now hold in cache, that will be generated by commands.  So the cache now goes beyond page fragments and pages, which is what you would typically have setip.  In the case of WXS this is data that could change frequently and have a lot of invalidations, and as we will see lots of invalidation requests are not great.

So rather than holding this data in the grid in this case we want some things to still be held at the application server level, but to do this we have to define additional cache containers.  It is worth noting that making this change will also help your standard Commerce environment with no WXS, especially if you are on V6, where this functionality was added later in the product.  It will be a reason why a lot of sites under V6 with cache will not have this enabled.

In order to do this the first thing that needs creating is a cacheinstance.properties file, you need to put that into the /stores.war/WEB-INF/classes directory in your Commerce deployment.  The file we have have created it looks something like this although it does an area to be tuned in terms of the number of cache entries.  We are creating two containers one general for the DistributedMapCache and the other for ListMemberGroupForUsers, but it can be split how you see fit.  When you look in the Cache Monitor you will see these containers.

cache.instance.0=/services/cache/WCDistributedMapCache
cache.instance.0.cacheSize=6000
cache.instance.0.enableDiskOffload=false
cache.instance.0.flushToDiskOnStop=false
cache.instance.0.useListenerContext=false
cache.instance.0.replicationType=1
cache.instance.0.disableDependencyId=false
cache.instance.0.filterTimeOutInvalidation=true
cache.instance.0.filterLRUInvalidation=true
cache.instance.0.ignoreValueInInvalidationEvent=true
cache.instance.0.disableTemplatesSupport=true
cache.instance.0.useServerClassLoader=true
cache.instance.1=/services/cache/com.ibm.commerce.membergroup.commands.ListMemberGroupsForUserCmdImpl
cache.instance.1.cacheSize=4000
cache.instance.1.enableDiskOffload=false
cache.instance.1.flushToDiskOnStop=false
cache.instance.1.useListenerContext=false
cache.instance.1.replicationType=1
cache.instance.1.disableDependencyId=false
cache.instance.1.filterTimeOutInvalidation=true
cache.instance.1.filterLRUInvalidation=true
cache.instance.1.ignoreValueInInvalidationEvent=true
cache.instance.1.disableTemplatesSupport=true
cache.instance.1.useServerClassLoader=true

With that file deployed then you need to add the content to define the data items into the cachespec.xml.  IBM have provided an example of what data to cache in samples/dynacache/cacheinstances.properties, so take the content defining the commands and copy it into your cachespec.xml, then deploy that update.

Changes to Custom Dynacache Entries

One of the reasons that things had appeared so slow when we enabled WXS was the amount of invalidation requests that were issues, had and the way they were grouped.  This is an important change you would not see locally because there is no overhead.  In the cachespec we had several different stores defined and they were all doing invalidations, on content within them.  It was just something which had built up over time as people had added content into the cachespec.xml file.

Because we listed all the commands that could cause an invalidation to run, what would happen is say we had a basket update the invalidations would run and eighteen different requests would be fired at the WXS environment.  Each one takes time and added together that was taking quite a significant amount of overhead on the request.  So we went back and updated the cachespec so that all the components we were invalidating had the same dependancy-id, and then cut down the invalidations to a single request.

On the dependancy ID that was defined on fragments such as the mini basket, previously each one had its own identifier.  So the dependancy ID was Minicart:storeId:DC_userid which is how the IBM sample file would define user based fragments.  Now all user based fragments in the stores say exactly the same for the ID we will use for invalidation and by doing that they are all removed in a single request on the store rather than multiple requests.

<dependency-id>StoreIDUserIDForInvalidation
<component id=”storeId” type=”parameter”>
<required>true</required>
</component>
<component id=”DC_userId” type=”attribute”>
<required>true</required>
</component>
</dependency-id>

Then after the commands that we would run invalidation such as add to basket, we define a single invalidation.

<invalidation>StoreIDUserIDForInvalidation
<component type=”method” id=”getCommandContext”>
<method>getStoreId</method>
<required>true</required>
</component>
<component type=”method” id=”getCommandContext”>
<method>getUserId</method>
<required>true</required>
</component>
</invalidation>

Once we did this we saw a major change in how the environment performed, again it would even help locally but the measurements would be much smaller periods of time than the remote requests.

Server Restart

Although changes to the cachespec.xml file will be picked up dynamically the other changes will require a server restart.  If your catalogue processes are running in WXS you should see it showing that the grid is active from inside of WAS also in the systemout log file for WAS will show the containers and connections being made.  If the WXS environment is not up and running I have noticed that your WAS server will just stop on startup if the grid is not available.  It does not throw and error it will instead appear to do nothing.

Part II

In the next part I will look at the changes we had to make in the WXS environment from the standard setup plus any other general tips we found useful.

*=info: com.ibm.websphere.commerce.WC_SERVER=all:
com.ibm.websphere.commerce.WC_MESSAGING=all:
com.ibm.websphere.commerce.WC_SENSITIVE_INFO=all:
com.ibm.commerce.foundation.*=all:com.ibm.websphere.commerce.WC_TRANSPORT_ADAPTER=all

That trace showed me the issue was within the businessobjectdocumentprocessor and how it was reading in the xml file that defined the service.  I looked at a trace that worked and a trace that failed.  After a quick Google I ended up on this page and after first making sure the authority to the files was OK.

www-01.ibm.com/support/docview.wss?rs=0&context=SSC8NWC&q1=null&uid=swg21386407&loc=en_US&cs=utf-8&cc=us&lang=en

I went and checked my wc-server.xml file and the section required for component services was not there.  That explained why everything looked good but it could not find the required files.  One I added that in I ran the updateEAR process, and restarted the servers after synchronising the changes out and things sprang into life.

It really was a lot of effort and all from a NullPointer error!

Bleddyn

Updating IBM Rational Developer for i to 7.5.0.x is not allowed

There is nothing mentioned about this in the info center about this, unless they read my feedback  it even still says update your update manager to install the fix.  Very likely as well you have your update manager set to check for updates.  So by the time you install the patch you will see the problem, you will look and if you are lucky you will find this link www-01.ibm.com/support/docview.wss?rs=&uid=swg21439225.

We ended up uninstalling the update manager and downloading the 1.3 version that they list in the fix page then installing the fix which did work, but its all a waste of time.  You waste your time trying to find the error, updating the update manager  uninstalling the update manager and reinstalling the update manager.  As for being told why don’t you update to V7.6 that will probably not work for a lot of people.  Also no consideration for users on low bandwidth connections, yes it would be great to do but that would be in a users plan.  or it is an interesting way to force users to update by breaking the normal process.

First of all I ran up a test environment a standalone 6.1 WAS instance at 6.1.0.31, Commerce at 6.0.0.10 and WebSphere Extreme Scale at 7.1.  Ok right now this is not supported but I could see no reason why it would not work.  We had the WAS/WC environment running on AIX 5.3 and the WXS was remote and installed under Ubunto 6.  We wanted to use the standalone environment for WXS to save us money (no WAS license), and create quite a simple configuration.

Setting up the WAS Side

The first step is to ensure you are patched to the latest levels of WAS 6.1  because of the Commerce migration I had done then the site was running 6.1

Restore the WXS installer onto the environment, and run the installer.

Setting up the Remote Extreme Scale Server

Catalog Server

The catalog

Catalog Container

Extreme Scale Console

The extreme scale console is new in WXS 7.1 and can be used to monitor the environment.

It is started from the <installpath>/ObjectGrid/bin directory using the command startConsoleServer.sh.  one thing I did notice under Ubuntu is that because the install of the software was performed under root, if I used the standard profile it did not now have authority to create the directory structure required under <installpath>/ObjectGrid/console where it wanted to create a number of objects.  This includes a Derby database and a number of files.  I have ended up using sudo to start the console server, but it will be something to address the simplest thing will be to set the right authority.

One the server is started in the browser goto https://<hostname>:7443, use admin and admin as the password and profile.     My first thought is this is quite a nice interface it looks good and the first step is to add the catalog sever.  Enter the hostname the JMX port, which should default to 1099 and the listener port 2809.  No connection will exist until you then create an Extreme Scale Domain.  At first I was unsure should go here because the 7.1 domain documentation only talks about WAS 7.1, but the domain name can be anything.  Once the domain name is added then on the right hand side of the screen add the catalogue server.  The status in the top right hand corner should now go to connected.  If you click around the Commerce site and then go into Monitors > Data Grid Overview you should see DYNACACHE_REMOTE, click that and you will get some details of what is taking place in the cache.

In the environment I am first setting up with a single catalog server and container, then I need to define