Data auditing in a replicated environment

One of the problems I have seen people struggle with is the ability to check the data contained in one file with the data stored in another. High Availability products have been supplying a audit suite to check this for some time with varying degrees of success.

RAP/400 is not a HA product but does rely on the replication technology (Remote Journalling) that these products use. RAP/400 is simply applying the data in a different manner by using the IBM commands shipped with the OS and not a self designed technology. But the question is still asked, is the data in sync between the two sites?

There are a lot of reasons for the data to be out of sync, the main one is user error, either they have restored the wrong data, the save was not done correctly or the user has inadvertently accessed the programs on the remote system and updated some data. The problem is how do you know which data is incorrect? Replication processes do not check the data as part of the apply process (this would be too inefficient) and only sense a problem when an update occurs such as a delete of a record which doesn’t exist, an add to the file occurs at the wrong RRN or the add fails because the file has less records than the write is attempting to write to. This is not to say errors are an everyday problem in most cases, but you wont know unless you check that the data in each record is the same.

We decided to look into this problem with a few limitations, the check will only look at the data stored with a DB file (not IFS based) and it will only fix data errors. If other errors such as RRN’s are out of sync occur we believe a full restore is the best option, the amount of CPU and resources required to fix up this kind of problem could be huge. We also wanted it to run solely on the target system, RAP/400 only installs on the target system so to fall in line with this we had to come up with a effective single system solution. We now have a simple technology test developed, when run it checks the data from each system record by record. If it finds an error, it will simply copy the data from the source system and write it over the target system data. Eventually we will make it store the updates and allow the user to determine which ones it should apply.

We expect this new option to be released in a future release of RAP/400 but if you are interested in understanding and testing the technology let us know.


Initial test show promising results for RAP/400 V1R1

Today we set up a couple of LPAR’s on the IBM VLP systems with RAP/400 and a data generation program. The program simply adds over 5000 records into a set of files over 5 libraries, it also adds data to Data areas and creates IFS directories and files. The environment is set up to automatically journal the Files in the file libraries and IFS directories and objects as they get created.

Installing RAP/400 took about 3 minutes and to configure the 6 journals took less than 2 minutes. So overall very easy to install and set up. In fact it only took us 5 hours to build the entire environment from scratch using the system commands so I would expect the general users to be able to do the same in less than 1/2 day because the data and objects will already exists as well as the journal environment.

The data generation program takes about 40 seconds to run on the source system for each library and we run it against the 4 libraries we have set up to contain the objects and files. The configuration has 6 journals which are all remote journalled to the target system. 2 journals carry the *IFS, *DTAARA, *DTAQ object and the other 4 have all of the DB files journalled to them. The program also deletes and creates a file in a specific library and adds entries to it after it has been created.

We run a program which changes the journal on the source system which in turn triggers the apply of the data on the target system. As soon as we call the program to change the journal we switched to the target system to view the system activity on the target, we were very surprised initially to see no activity at all. Thinking the apply had not occurred we check all of the objects and found that the apply process had in fact run, so fast we never saw the activity. A couple more attempts and we did see the process run for a few seconds, we feel this is pretty amazing considering we are processing 6 journal receiver requests through the process.

The next step will be to increase the number of file adds and see what the impact is, the CPU utilization was minimal as well.

I have been demonstrating HA products to customers for a number of years and was always able to capture activity through the apply process. This is going to be more difficult to show.

I hope to have the next version available for shipment at the end of May 2007 which is a major improvement on the previous release.

If you would like to trial the new product and have a fairly heavy processing box let us know, we would like to see the product working in seriously hard working environments.


C programming for the i5

The recent article about how RPG is seen as the only programming language for the IBM System i5 raised quite a few comments from readers. The article was more about how other languages are supported on the i5,yet as a group of i5 supporters and the media in general only ever talk about RPG. I know PHP and Java etc have been gaining some coverage of late but the majority of articles still talk about RPG and how to do interesting things with it! I still see articles on how happy we are that some college has decided to add RPG and the i5 to it IT courses. Lets no kid ourselves this is not going to encourage people to use the platform more. One respondent said they would never use anything but RPG for business logic as the others languages such as SQL and PHP Java etc. are not good enough for business logic??

In an attempt to help others see alternatives to the RPG language for crtical business logic I hope to produce a few code snippets that show how to do things such as read files, call API’s. create Interfaces using UIM etc. The code will eventually build into a tool for others to improve upon, hopefully others will post enhancements and new code snippets. This will not turn out to be an open source revolution, but it may encourage some to view the alternatives.
Couple of things I have in mind are

  1. Simple file reading writing and content display in C
  2. Read through a Journal
  3. UIM panel display
  4. Disk utilization
  5. Profile replication

If you have any suggestions for the kind of tool you would like to see produced let me know and I will see if it makes sense to build it for others to use.


RAP/400 ongoing development

The RAP/400 Version product has been out there for a few weeks now and has seen a fair level of attention from the media and interested parties. Some of that attention did amaze me, I have been a HA specialist for a long time and was very surprised to hear from a number of the large HA product vendors that they felt I was now competing with them?

RAP/400 is not a HA product, it provides support for journalled objects only and does not provide many of the features that are required for a product to be considered a HA solution. The main reason for RAP/400 was to provide the users with an alternative to the HA products where those solutions were just too heavy and over kill for what the customer requires. During my time as a HA distributor and working for one of the HA providers I came across a number of situations where the customer wanted a replication tool just to protect their data. The only option these people had was to buy the HA product (or DR product if available) and make the best of it. I am looking to fill that gap, giving people the choice of a low priced solution which only protects those objects which can be journalled. Running a HA product requires a reasonable level of skill, managing that product after the install has been completed also requires skill and attention. If you just need to protect your data on an ongoing basis why go for a solution which while providing a lot more options brings with it the management overhead you don’t need?

I am now well into the next level of development, not that customer have been flooding to the door pushing orders upon me, but because I can see that some of the limitations I set in the initial release should be catered for in a normal users environment. IBM is providing all of the technology, I am just putting a wrapper around that technology so users can set it up easily and have little to no management tasks. IBM has always provided good plumbing but sometimes fails to provide the interfaces to that plumbing for the user, we have done that with the RJ technology and the apply process. We only need to run on the target system and have no requirement to run any programs on the source, IBM is doing that for us already. This release will support up to 100 journals, each will be either supporting *IFS,*DTAARA and *DTAQ or *DB. This allows the product to support the DDL (File level changes) which was a major problem in Version 1.0. I have also improved the interfaces for configuring and managing the product. Errors can now be viewed through a number of UIM panels instead of having to use Query Manager.

Another change will be the licensing used by the product. IBM has recently announced the availability of the i515 and i525 servers, these servers have a new licensing policy which is based on users not P groups. RAP/400 will be an ideal companion for these servers and will be priced on the number of journals which are configured. The new release should be available mid June for shipment although we will require a number of beta installs before then.

I feel the product fills a gap in the market which has been left open for a long time. The HA solution is too heavy to fill this gap and users want more than just tape saves, they need to be able to have their data protected in as near real time as possible. Recovering the system and using that data immediately is not necessary. The product will allow the user to determine when the data is applied just by controlling the change of the journal receiver on the source, so if you need to have no more than 1 hour of data waiting to be applied you can just change the receiver every hour. IBM does the apply of the data so I believe its going to be much quicker at applying those changes than any process I could write. Batch runs can be protected, just change the journal receiver at the start and end of the batch process, if the batch fails the changes have not been applied to the target.

If you would to be involved in the beta testing please visit the support pages on our website where a link to the beta request page is located.

Using C on the i5

I recently wrote an article for the System iNetwork iDigress publication about how the i5 seems to be stuck in a rut because the rest of the world thinks its only RPG that developers use on the platform. You can see the article here Article
There are a number of responses to the article which can be found here but I also had a number of people write directly to me. The main responses were about how right the basic assumption that people only relate RPG with the i5 was but some did think the article was saying RPG should not be used! Thats not what I thought I wrote but at least there was a dialog, I feel there is a place for all languages on the platform as long as they are supported. The i5 supports a plethora of languages and the strengths of each have to be understood before you decide which elements should be written in what language. But if the industry (the media have as big a part in this as IBM does) don’t start to publicize the act that the i5 does support all these other languages instead of just pushing RPG in every article we will never get the traction in the developer community we need to succeed. Just thinking Java is the answer to all the problems is also a problem, the language is never going to be as efficient as RPG, C, Cobol and the likes, but it is good at providing a level of portability and code reuse so if thats important you should use it.

I stated that I have never written a single line of RPG code yet still have a number of successful products running on the platform. These products are required to run very efficiently and use a number of API’s and TCP/IP networking functions so C was the better language for me to use. I wrote an application for a major UK bank a number of years ago which again was written in C, the bank initially said it had to be written in RPG but agreed after initial trials to it being fully written in C. When questioned why they wanted RPG it was because they didnt understand the support the AS/400 (as it was then) had for the C language. That support has grown over the years whhich has allowed the language to compete for application components that require DB access, this is something RPG was king for previously.

I hope the debate continues and people start to see how the i5 can provide many benefits to its users and developers alike.