Mar 25

JobQGenie Version 5.1 is now GA

We have been very busy getting this release of JobQGenie and a new Version of RAP out to the customer base which is one of the reasons we have not been posting much on the Blog! But we now have Version 5.1 of JobQGenie up and available for download from both the download page of the Blog and the Associate member pages on the website.

This version of JobQGenie has a totally new data capture and storage engine built in. Initially we started to look at a low level solution to be packaged with the RAP product that would offer similar technology but at a much higher level and with less functionality. After some deliberation we decided to completely rewrite the capture technology as it would be far easier than trying to bring elements of the very complex capture technology we had in JobQGenie into RAP, this proved to be a very worthwhile exercise as we were able to change some of our original concepts and use much faster and newer API’s. The new process turned out to be more efficient and provided us with a 100% capture rate for any jobs which went through the job queues.

We had been struggling with one of our current clients due to the very complex and intensive environments they run (they readily accept that they stress all of their suppliers products beyond what most others do) which caused the data capture process to run into timing issues causing some data to be lost of inadvertently overwritten by a slower process. They needed something which could handle many thousands of jobs per hour without missing a single job and yet still provide an efficient collection process. We tried to get IBM to give us more access to the job data but any changes they agreed to would only be implemented in a future release of the OS, this customer needed the solution now! So we took the new technology from RAP and started to add the same functionality we had provided in the previous versions to it.

The customer has been working with us on getting the solution tested and we have finally manged to get to a stage where they are happy with the results and hope to move it into production as soon as possible. We still have some final testing before we get the final sign off but this is the best results we have seen so far with no missed data at all! They understand that recovery from an unplanned outage is not going to be easy without this piece of the puzzle which is why it is now seeing a lot of attention paid to it.

The data capture is only part of the story, once you have the data you need to be able to work with it and determine what jobs affected what data and how that data can be rolled back to a suitable starting position. The tools provided with JobQGenie allow you to take a job and look in all of the Journals on the system to see what if any data was affected by the job. This is very important where you are using a High Availability tool for data replication which applies data as it is received. The apply process only knows about the data and where it needs to apply it, not how it was created and when. If you have to restart a job, you need to know if any data needs to be rolled back before it is restarted otherwise it will either fail of corrupt the data itself.

After you have cleaned up the data, the jobs can be resubmitted using the data collected from the source system, in some cases this will allow a batch stream to be started mid stream instead of from the beginning. A simple option from the list of jobs is all that’s needed to re-submit the job.

JobQGenie was written to work with any of the HA solutions and requires little configuration to start the capture process, replication of the data to the remote system requires configuration of the files into the HA solution which should take less than 1 day to complete the install. If you need any help we can provide remote support at very low cost to the user.

Please feel free to contact us about any question you have regarding JobQGenie or any other product we support.

Chris..

Mar 25

RAP Version 4.1 is now GA

We have finally finished the testing of RAP Version 4.1 and have placed it on the website for download. The product has many new features which will allow us to compete on a level footing with other HA products. Previously we had always described RAP as a Data Replication tool but with the new features we have added we now feel it is more akin to a High Availability Tool than the Data Only Replication tool it started from.

If you are interested here are a few of the new features.

We have improved the data apply process and provided a more detailed view of the replication as it occurs. While we still cannot show the actual journal entry being applied (I question why you need to know that anyhow) we do show the current block of journal entries the process is working on. New functionality has been added to show other replication status as it occurs as well while trying to keep the management requirements to a bare minimum.

Object replication was something we steered away from, we feel the most important asset any company has is it’s data, objects tend to be fairly static in terms of change activity while the data is in constant change. This release has added automated object replication to the mix, we use the audit journal to determine when objects change. Once we find an entry we will determine if the change can be carried out by a simple command or if we have to run a save and restore process to bring the source and target objects into sync. As we move forward we see this as one area where we will continue to provide additional improvements to cover additional objects and filtering.

Job Queues are important for recovery, if you don’t know what jobs were running and how they ended you will have to spend a lot of time doing data analysis to find a restart point for your applications. With JobQGenie we provided a very granular process to capture a jobs life cycle as it went through loading to a job queue, when it became active and how it ended, RAP has less granular version of the technology built in but still using the latest capture technology. In fact we built the new capture technology for RAP and migrated it to JobQGenie because it was so good! As part of this option we had to build a data apply process which uses the journal entries in the remote journal and applies them directly to the database, this is a first step into bringing an alternative apply method to RAP which is the same technology our current competition uses. The foundation is there to be built upon and we will continue to improve and add functionality to this technology for future releases.

Sometimes it would be very nice to see what jobs were active prior to a system crash, the new Snapshot process allows the active job data to be captured and sent to the remote system on a regular basis. Should a system failure occur you can start up the snapshot browser and view the data in the same manner as you would using the WRKACTJOB command.

iASP technology is starting to take hold with more and more companies looking at its capabilities. RAP has supported iASP level replication since Version 3.1 and this release adds further functionality. One area we felt this technology would be useful is in a recovery center where many customers can be set up in dedicated iASP groups to ensure data segregation without the need for LPAR level configurations.

As you can see this release adds major new functionality to the product, yet the price structure will not change to that of the previous level. New functions are already on the drawing board to support Output Queue replication and other objects. We will continue to add new features to the Job Queue Replication option and have a number of new interface options under consideration, RAP is already a very capable replication solution and as the new features are added we are sure its popularity will grow.

We have added download links to the Blog under the download tab for those of you who have registered here. For those who have associate memberships for the Shield website we have also added the relevant objects and links there. The product installs with a 30 day key which gives you time to install and test some of the functionality prior to buying. There is no obligation to purchase so download a copy and give it a whirl!

Chris…

Mar 12

JQG Authority Issue

When we repackaged the latest Version of JQG (Version 5.1) we found a problem with the authority to the library and the Data Queues. This is a problem we asked IBM to fix which they said no to!

The Library has to be set to *USE and the Data Queues set to *CHANGE when used on Version 5.4 of the OS onwards. IBM has changed the way they load the data queues for some reason using the submitters profile instead of a system profile. It only affects the JOB LOAD Data Queue but we have taken the decision to change all of the Queues to ensure we don’t have problems later when IBM does the same thing to the other queues.

A new copy of the program will loaded to the Blog downloads later this week, until then if you have downloaded the product ensure you change the authorities to the above or the process will fail to register the job activity.

Chris…

Mar 10

RAP Version 4.1 ready for packaging

We are excited with the new features of RAP and have finally started on the packaging stage after extensive internal testing of the new features and there interaction with existing ones.

Packaging is always a time intensive process as we have to get the installation programs working correctly with the IBM RSTLICPGM process. Lots of times we have to rebuild the entire product from scratch even for minor program changes. The reason for the rebuild is because we cannot test the exit programs without running the RSTLICPGM commands, this means we have to package the product entirely just to see if changes made have the desired effect.

While the process could be laborious we have written scripts and use the Software Development Manager product to great effect. Once we have a successful install we then have to configure and test the processes again before we test the DLTLICPGM process to ensure we clean everything up correctly.

We think we are still on track to meet GA by the end of the month.

As soon as we have the product finished and the features fully tested we will give out a list of what the product provides.

Chris…

Mar 04

HA on a shoestring PartII

I finally found the links for the programs for the programs IBM used in the Lab sessions for HA on a Shoestring. Here they are for those of you who are interested.

http://www-03.ibm.com/systems/resources/systems_i_software_db2_chgnsleep.cl.txt

http://www-03.ibm.com/systems/resources/systems_i_software_db2_haapply.cl.txt

http://www-03.ibm.com/systems/resources/systems_i_software_db2_hrtbeat.cl.txt

http://www-03.ibm.com/systems/resources/systems_i_software_db2_hrtmonitor.c.txt

http://www-03.ibm.com/systems/resources/systems_i_software_db2_hrtstatus.c.txt

Enjoy…

Chris…

Mar 03

IBM’s HA on a shoestring Presentation

We seem to be getting many requests for information on our RAP and the link to the IBM HA on a Shoestring presentation which has made the rounds at Common a number of times. We produced a white paper many years ago which was the starting point for the technology, it only showed how we took a journal receiver from a remote journal and applied it to the database which was attached to a local journal but it did show what was possible. As you can see from the presentation we did work very closely with Larry in making the technology work, Larry was an inspiration and was madly committed to the Journal Technology, we do miss him a lot!

RAP has changed significantly since those first discussions with Larry Yougren, but the basic concepts are still the same. We have listed the presentation in the Downloads section of the Blog for those of you who didn’t get a copy from attending an IBM session but want to understand the principles.

If you want to create your own HA on a Shoestring program the advice given in the presentation is a good base to start from. We had to make significant changes to the processing due to the APYCHGJRN and APYCHGJRNX incompatabilities and have added many features which make the product much more user friendly and probably closer to true HA than the HA on a Shoestring will ever get.

A couple of things to remember if you are going down this path

1. The APYCHGJRNX command is required if you want to support DDL entries (File creates). However it does have some features missing which only APYCHGJRN provides. You cannot run the two commands against the same receiver in a two pass process! V6R1 removes the differences so if you have V6R1 it will be a lot simpler. The other option is to use the QDBRPLAY API to get over the APYCHGJRN functional deficiencies.
2. Data Area and Data Queue’s have no creation support built in and manual processes to build them and apply the changes on the fly don’t work! You will have to come up with an alternative option if this is required.
3. The HA on a Shoestring programs do not cover error reporting in any depth, you will also need to create management functions for the journals on both systems.
4. Auditing is not provided nor is any switch management so be prepared to build those as well.
5. There are a few gotcha’s which are not made clear in the presentations. Read the Journal manuals closely….

I did know where the IBM source was for the programs but cannot find it to add to this post. If I do find it I will post it for anyone who is interested.

If you get stuck don’t expect us to help, we are only making the presentation available for those of you who have the skills to take this further… If you want to trial RAP you are free do do so.

Have fun!

Chris…

Mar 01

Latest Version of JobQGenie

We have taken the new collection engine which we developed for RAP and built it into the latest version of JobQGenie.

The new collection engine was needed due to the older technology becoming too cumbersome to manage, every time we needed to make a slight change to the process we had to make changes to many other programs to make it work. The newer technology is more modularized and has a better tolerance for high volume environments where jobs are being loaded at a rate which would cause timing issues before.

We have also asked IBM to supply better sequencing options for the data collection which will be easier to implement in the new engine. Unfortunately these changes will only be available in future releases. The new product has simplified menu’s and display panels which are more intuitive than the previous versions. The RAP product only provides the data collection and display process, JobQGenie takes this further by allowing the user to drill down and see affected data by Job so you can identify what recovery actions to take on your data before starting the jobs up again. The displays also show you jobs which could affect data recovery plus any objects which have been touched as long as they are journalled.

Here are a few sample screens of the product in action

View of Active Job Collection

View of Active Job Collection

A list of the journals which contain changes created by the job

A list of the journals which contain changes created by the job

A list of Journalled Objects which the job changed

A list of Journalled Objects which the job changed

Journal Entries generated by the Job

Journal Entries generated by the Job

The command string used to load the job

The command string used to load the job

The product is available for download from the download section if you are interested in trying the product out.

If you have any questions or need more information please contact us.

Chris…