Oct 24

PowerHA and LVLT4i.

We have had a number of conversations about LVLT4i and what it offers to the Managed Service Provider(MSP). As part of those discussions the IBM solution PowerHA often comes up as it also uses iASP technology but that is really where the similarity ends.

PowerHA uses the iASP to isolate the objects that are to be replicated to another system/storage device and it has an exact copy of the iASP from the source on the target. Changes are captured at the hardware level and are sent to the remote system as they occur.

LVLT4i only replicates objects to a remote iASP, it uses either Audit journal triggers or the Remote Journal technology to capture and send the data. The source object resides in *SYSBAS and the target object in an iASP, it is used primarily to allow multiple copies of the same library/object combination to be stored on a single system. The remote iASP is always available to the user.

iASP is not widely implemented at customer sites, this is in part due to the lack of support for iASP’s built into many of the applications that run on the IBM i today (many of the applications were built before iASP technology was available). For a customer to migrate an application to allow iASP use there are a number of constraints which have to be considered plus each users environment has to be adjusted to allow the iASP content to be used (SETASPGRP etc). This has further limited the use of iASP as many do not feel the benefits of moving to the iASP model out-weight the cost of migration. Another issue is you are now adding an additional storage management requirement, the iASP is disk based which will require protection to be added in some form. With LVLT4i you can leave your system unchanged, only the target system is going to need iASP setup and that will be in the hands of your Managed Service Provider. The decision about what to replicate is yours, with some professional help from a Managed Service Provider who knows your application it should be pretty bullet proof when it comes to recovery.

If you implement PowerHA you are probably going to need to set up an Admin Domain, this is where any *SYSBAS objects such as system values, profiles and configuration objects are managed. in LVLT4i we do not manage system values or configuration objects (configuration objects can be troublesome especially with TCP/IP) or system values. We have however just built in a new profile and password process to allow the security aspects of an application to be managed across systems in real time. Simple scripts can capture configuration and system value settings many of which are not important to your application so LVLT4i has you covered. If we find a need to build in system value or configuration management we will do so fairly rapidly.

PowerHA is priced by Core, so you license it for each Active Core on each system. Using CBU licensing, PowerHA can utilize lower active cores on the target and only activate them when the system is required. Unfortunately in a HA environment you are probably switching regularly so you will have the same number of active cores all the time. LVLT4i is priced by IBM tier regardless of the number of active cores. The target system license is included with the source system license regardless of the target system tier so a Manage Service Provider who has a P30 to support many P05 clients is not penalized.
PowerHA also comes in a few flavors which are decided on by the type of set up you require. Some of the functionality such as Asynchronous mirroring is only available in the Enterprise edition so if you need to ensure your application is not constrained by remote confirmation processing (waiting for the remote system to confirm it has the data) your are going to need the Enterprise edition which costs more per core. LVLT4i comes in one flavor and is based on a rental model, the transport of data over Synchronous/Asynchronous remote journals is available to all plus it supports any geographic model.

Because the iASP is always available the ability to backup at any time is possible with LVLT4i. With PowerHA you have to use a Flashcopy to make another disk based copy of the iASP which can then be used for the back up to tape etc. That requires a duplicate set of disks to match the iASP content. With LVLT4i you can use Save While Active or suspend the apply process for point in time saves, the remote journal will still be receiving your application updates which can be applied once the save has completed so data protection is not exposed.

RPO is an important number which is regularly banded around by the High Availability providers, PowerHA states it is 0 because everything is replicated at the hardware level. We believe LVLT4i is pretty close to the same but there are a couple of things to consider.

First of all, RPO of 0 will require synchronous delivery of changes, if you use an Asynchronous delivery method queued changes will affect that for either solution. LVLT4i uses Remote journalling for data changes, so if you use Synchronous mode I feel the two are similar in effect.

Because we use a different process for object changes, any object updates are going to be dependent on the level of change activity being processed by the object replication processes. The amount of data being replicated is also a factor as a single stream of object changes is used to transfer the updates. We have done a lot of work on minimizing the data which has be be sent over the wire such as using commands instead of save restore, pipe-lining changes so multiple updates to an object are optimized into a single action and compression within the save process. This has greatly reduced the activity and therefore bandwidth requirements.

PowerHA is probably better at object replication because of the technology IBM can access, plus it is going to be carried out in line with the data changes. The same constraints about using synchronous mode affect the object replication process so bandwidth is going to be a major factor in the speed of replication etc. Having said that, most of the smaller clients we have implemented any kind of availability for (HA4i/DR4i) do not see significant object activity and little to no backlogs in the object replication process.

The next recovery figure RTO talks about how long it will take from making the decision to switch, to actually switching. My initial findings about iASP tended to show a fairly long role-swap time because you had to vary off the iASP and then on again to make it available. We have never purchased PowerHA so our tests are based around how long it took to vary off and then on again a single iASP on our P05 system (approximately 20 minutes). I would suspect the newer and faster systems have reduced the time it takes but it is still a fairly long time. LVLT4i is not a contender in this role because we expect the role-swap times to be pretty extended (4 – 12 hours) even if you do a lot of automation and preparation.

One of the issues which affect all High Availability Solutions is the management of batch, if you have a batch process running at the time of failure it could affect the integrity of the application data on the target system. LVLT4i and PowerHA both have this limitation as the capture of job queue content is not possible even in an iASP, but we have a solution which when integrated with LVLT4i will allow you to reload job queues and identify orphaned data which has been applied by a batch process. Our JQG4i product captures all activity for specific job queues and will track each job from load to completion. This will allow you to recover the entire application environment to a known start point and thereby ensure your data integrity is maintained. Just being able to automatically reload jobs that did not run before the system failure is a big advantage that many current users benefit from.

There are plenty of options out there to choose from but each has its own strengths and weaknesses. LVLT4i uses the same replication technology as out HA4i and DR4i products with enhancements to allow the use of iASP as the target disk. It is not designed to meet the same RTO expectations as PowerHA even though both make effective use of iASP technology. However, PowerHA is not necessarily the best option for everyone because it does have a number of dependencies that make it more difficult/costly to implement than a logical replication solution, you have to weigh up the pros and cons of each technology and make a decision about what is important.

If you are interested in knowing more or would like to see a demo of the LVLT4i product please let us know and we will be happy to schedule.


Jun 03

VIOS and our new Power7+ system.

When we ordered the new Power 720 we had always planned to partition it up and have multiple IBMi partitions running, we chose to go the IBMi hosting IBMi route as it was the simpler of the options we had available. Now with the new Power8 systems IBM is recommending that we brush up on our AIX skills (YUK!) and look at using VIOS as the hosting partition as this will be the way of the world… So we are going to bite the bullet and remove the existing IBMi partitions and replace with a brand new configuration using VIOS.

As part of this change we are also going to install AIX and Linux partitions, these are mainly going to be for testing but as we use Linux a lot for our web development it will allow us to move our production Linux servers to the Power7 system. The IBM i partitions will be running under VIOS as well which will remove the minor headache we had of having to end the hosted partitions while we did maintenance on the main hosting partition, this is our main development partition so it was the one where most of our daily activities occurred and is kept up to the latest PTF levels often.

We have downloaded a couple of red books and red papers as part of our planning which we will use as a guide to setting up the system, having looked at the content we will certainly get a refresher in AIX command line processing as we move forward. We have also contacted IBM about our processor activations as it looks like it was screwed up when we purchased the system and subsequently added an additional IBMi activation. Eventually we should have 2 IBM i cores and 1 AIX core activated (not sure about the Linux activation but it should run as a micro partition using the AIX activation?) so we will micro partition the 2 IBM i cores across 4 IBM i partitions and have either AIX and Linux or just Linux running on the additional core.

The first thing we are doing is doing a system save of all of the partitions, the save of the hosting partition will actually save the hosted partitions but for installing under VIOS we will need the saves of the individual instances. When we restore the main partition we will need to somehow remove the hosted partitions (not sure how we restore the system without the NWSD objects and configurations but I am sure IBM will have some answers).

Once we have saved everything we are going to need to delete the existing set up and create a new drive configuration (currently raid 6 protection set on all drives) because VIOS needs to be installed on a separate drive and we want to set the drive protection at the VIOS level for the remaining drives (at least that’s my initial thoughts).

As we progress through I will be posting updates about what we have achieved and some of the problems we encounter.


Sep 04

FTP Guard4i gets new feature

One of our clients was interested in the FTP Guard4i product and wanted to secure their FTP environment from unauthorized access. We installed the product and set the security so that all FTP access would now be monitored and restricted. Unfortunately after a few minutes we had to turn off the security because the client had not understood just how much FTP activity was carried out on his system. This was a problem because they did see some attempts to access the system using FTP from unauthorized users yet they could not identify all the authorized users until they hit the site and were rejected by the security settings. At first we were just adding users as they showed up in the log after checking that they were in fact authorized, but that gave a number of issues because the FTP access used by the users was not built to recover when the request was rejected. So we eventually turned off the security and left it up to the normal object security to handle the issues until we came up with a solution.

This concerned us as we did not like the fact that FTP activity was going on and the client was unable to see just how bad the problem was. So we started to think about how we can show the problem exists while not affecting the existing processes. Eventually we made a change to the programs that would allow the security to circumvented while still logging exactly what and who used the FTP services. Now the client is able to see all activity and we can build the FTP security using the log information before implementing the fully secured environment.

FTP is very unsecure and should be turned off where possible, if you must have FTP services turned on we suggest you investigate the installation of a security and logging package such as our FTP Guard4i. Just understanding the level of FTP activity that is going on could help you determine just how exposed to data theft you are.


May 16

Pagination now added to log viewer

One of the tasks we left out in the initial release of the PHP Interface of FTP Guard4i was the ability to set the page size when viewing the log entries. What we wanted to do was allow the number of log records displayed to be preset by the user, this would allow the retrieval of records to the page to be carried out a lot quicker than if all of the records were to be displayed. As part of this exercise we also decided to add a search button for data stored in certain columns of the database, this would allow you to say filter the records based on a certain object or on a certain user etc. and still provide a paged output.

The following is a sample screen where the sort parameter is the date and time column, because we provided the sort capability we do not need a search capability as well so no search box is displayed.

Paged Log View

Paged Log View

Here is a sample screen showing the sort column being the Object information and the search value was QSYS.

Paged View with Search

Paged View with Search

We are constantly looking at ways to add new features and functionality to the FTP Guard4i product, if you have any questions or would like to see a demo please let us know.


May 06

FTP Guard4i is available for download

FTP Guard4i is now completed and available for download. We have placed the manuals online as well as the objects required to install the product. You will need to sign in as a member to download the objects and once installed you will need a key to allow the product to function. The PHP interface is available and requires the Easycom i5_toolkit functions to allow connectivity to the IBM i. We have not tested it with the Zend Free toolkit at this time and would need to make some additional changes due to the lack of support for some objects. If this is needed we can work with you to make those changes.

FTP Security is something we have been looking at for a long time, our initial requirement was highlighted because of the access to the source code for our products by the developers. We needed to give them access to the code to allow them to carry out their activities but we did not want them to be able to copy the code to other systems. The original product we created also provided an FTP Client so we could make the object transfer a lot easier than the FTP Client provided by the OS but this release only provides the security aspects required.

As part of the rewrite we have made a number of improvements in the methods we used to control the access particularly around the accept and reject IP addresses set for individual users. This allows you to set a range of IP addresses a user can connect to and from in the same manner as you can set the connection accept and reject addresses. We have also changed the logging to a Database file which allows us to add much more meaningful data about the activities carried out. While the clean up routines we have provided only allow the log to be cleared, using standard SQL against the file will provide a lot more granular entry removal.

FTP Security is an area most IBM i shops ignore because they believe the IBM i is naturally more secure than other platforms, that is not true and as we see more and more IBM i systems being linked to a wider audience we could see more intrusions being logged. FTP Guard4i also has a very comprehensive logging feature so you can now see who connects to your server and what they did while they were connected.

If you need more information about FTP Guard4i or would like to see a working demo please let us know using the demo request forms on the website.


Apr 29

FTP Guard4i interfaces completed

We have finished the PHP interfaces for FTP Guard4i. The 5250 interfaces are going to remain pretty much the same due to the limitations set by UIM (80 columns does not fit all of the data) but we hope to eventually add some new screens once we work out what makes sense. The PHP interface uses the i5_toolkit functions to extract the data from the IBM i, this allows us to run the Apache server on a separate server which is better suited to running an Apache web server than the IBM i. We also have the same processes running under iAMP on the IBM i for testing and demonstration purposes if you wish to see a total IBM i implementation.

Here is a quick overview of the pages and the data that they show.

1. FTP Guard4i Status screen

FTP Guard4i Status

FTP Guard4i Status

The list of users who are connected to the FTP server is a new feature which is only available in the PHP interface for the initial release due to the limitations imposed by the UIM (5250) screens. We did some testing with multiple users to see exactly what users were logged in and when which provided some interesting results.
The FTP Server is the job which is listening on port 21, the SSHD Server is the job which is listening on port 22. The log writer is the job which processes all of the request events which have been created as a result of user connections, this data is stored independently so even if the log writer is not running the events will be recorded waiting for the log writer to be started. We have also listed the exit points which have been correctly registered for FTP Guard4i, if any of these exit points are inactive no FTP activity will be logged until they are reset and the FTP Server restarted.

2. FTP Guard4i Server Users

FTP Guard4i Server Users

FTP Guard4i Server Users

Access to the FTP Server can be limited in many ways, the above image shows all of the configuration aspects of the users who are allowed to access the FTP Server and what limitations if any are set for that user. You can directly control all aspects of the FTP Server activity for a particular user such as when the can connect and where from, you can determine if they can move around the library/directory structure or if they are jailed to a specific one. If a user tries to connect to a directory/library which they are not allowed they will automatically be connected to the default directory/library. The list format and Name format are set regardless of the actual FTP Server settings.

3. FTP Guard4i Client settings

FTP Guard4i Client Users

FTP Guard4i Client Users

The FTP Client which is available on the IBM i is generally open to all users, this can be a major security exposure as a user with sufficient access can link a FTP Server to the system (a PC running FileZilla Server or similar) and transfer objects off to the PC without any trace. With FTP Guard4i all FTP activity is logged and can be reviewed to see what users did when using the services. The controls provided can limit the target Server (IP Address) and what activities the user can carry out, including the directory/libraries which can be accessed.

4. FTP Guard4i Accept IP Address

FTP Guard4i Accept IP config

FTP Guard4i Accept IP list

You can set the addresses which the users can connect to the FTP Server from, this is in addition to the IP addresses which can be set in the User settings which can provide a very simple to manage access tool. The process will check for an accept address and reject address entry, if an entry matches a specific accept entry the connection will be allowed even if a reject entry matches which is less specific. The User settings are checked after the connection to verify the user can connect from the IP address after this check.

5. FTP Guard4i Reject IP List

FTP Guard4i Reject IP

FTP Guard4i Reject IP List

The above shows a single entry which states that everything is rejected which does not match an Accept entry.

6. FTP Guard4i Log

FTP Guard4i Log

FTP Guard4i Log view

The level of logging can determine what log entries are placed into the log, if it is set to log all entries you will see an entry for every request made to the server including the actual files and directories which have been involved. This can be very important for auditors who need to view all of the transactions a user carried out via the FTP Services on the IBM i.

7. FTP Guard4i Config.

FTP Guard4i config

FTP Guard4i Config

There are various control files which determine how FTP Guard4i runs, the PHP interface provides the ability to view or update those files.

As you can see FTP Guard4i is pretty much completed, all we need to do now is carry out some additional testing before we move to the release stage of the process. We will also provide a manual which will give more details on the various configuration parameters and how to manage the data which is logged.

If you are interested in FTP Guard4i and the security of the IBM i FTP Services let us know. We can provide online demos of the product and show how effective it is in locking down user FTP activities. Don’t wait until your data has been stolen, act today and give us a call.


Apr 24

FTP Guard4i Log Viewer

As promised we have now developed the log viewer which shows the events which have been logged by the FTP processes. The log view has a number of columns each of which is sortable but the default sort is done by the Date and Time with the latest entry at the top. Here is sample view of the log on our test server.

FTP Guard4i log view

A sample of the events logged by FTP Guard4i.

A couple of interesting things came about while generating the log, you will see that we deleted a file ‘/home/CHRISH/??_???????`%??>?>????????’, one of the issues we all come across from time to time is where a file in the IFS has a strange name, deleting the file using the normal IFS commands is not possible as it will always return ‘File not found’ errors. Using FTP (actually we used FileZilla) you can see that we successfully deleted the file in question. The log also shows a ‘Send File’ operation, that was actually a get operation from the FTP client but the event gets logged as a ‘Server Send File’ operation..

The PHP interface is now pretty much complete but we need to do some more work on the UIM interface to align the data store with the actual output to the UIM Manager. Once that is finished and we have done some more testing FTP Guard4i will be available for download.


Apr 23

FTP Guard 4i Take 2

We had been discussing the FTP Guard 4i with a prospect and they mentioned that they would like to be able to monitor the FTP Server and SFTP Server from the FTP interface. So we have added a couple of new features to the status screen that allow the user to administer the FTP Server and the SSHD server which is used for the SFTP connections.

Here is the new status screen

New FTP Guard 4i status screen

FTP Guard 4i take 2

One of the things we did notice when we added the new features and checked they functioned was the SFTP connection takes on the QSECOFR profile in the job and drops the original user profile. We need to take a look at this to see exactly what effect this has? We don’t allow the QSECOFR profile to connect via FTP or SFTP so the security we have set for the user as far as FTP is concerned still applied.

Let us know if you are interested in this kind of solution and what if any additional features you would like to see. The Log viewer is coming along and will be the subject of our next post.


Jul 16

How to see what locks are against an IFS object.

I had been looking around for a method to find out who was locking a specific IFS object, the problem came to light after some new programs we were developing crashed before closing the files (I thought when a program ended the file would be automatically released, but it looks like that may not be so with IFS and abnormal termination?) which resulted in the objects not being replicated by our replication software. There is no WRKLCK command or WRKIFSLCK etc so we needed a solution.

After some reading through the various API’s to determine what might be useful we did a quick search on Google. This came up with a very interesting solution, the article pointed to the QP0FPTOS API which is described in the manual with no reference to the call parameters the article suggested. So not being one to shy away we decided to give it a try. The command we ran was “CALL QP0FPTOS PARM(*LSTOBJREF ‘/home/ha4i/log/HA4I409_debug.dta’ *FORMAT2)”. This was pretty confusing because neither *LSTOBJREF was listed or the *FORMAT2 in the documentation! But it works! So if like us want to see what locks are against a particular IFS object give it a try! Just change the object to be checked of course :-)..

The output is directed to a spool file which can be viewed, we had hoped to be able to get the data back in soft copy to allow us to take some action, but for now we at least have some visibility of who is locking the object. Maybe we will ask IBM for more information on the API and how else it can be used..


Dec 06

New Password # tool.

One feature of passwords is that they are encrypted so you cannot pull back a password to see if it is the same across a couple of systems. As part of the HA4i product we replicate passwords between systems using the encrypted blocks of data we can retrieve using API’s, we can also compare the two passwords by checking that the encrypted block is the same on each system (It should be). Recently we had a problem where a password would constantly return an error when we checked the data returned by the API so we needed a separate process to allow us to see what the content is on each system when we change passwords via a save and restore and update method. This has resulted in a tool which allows the CRC which we build using the data returned via the API to be seen by the user on each system. The tool is available for download from the downloads page.

If you have any questions or concerns with using the tool let us know.