Feb 29

POP3 email is not working

I have logged a problem with the ISP because I have seen no email since last night. If you have sent Email and not had a response its not because I am ignoring you! As soon as they fix the problem I will get back to you. I did run some tests and its definitely not getting through, even to new addresses I built so it could be an MX DNS entry problem somewhere?

Chris…

Feb 29

Declaring a LONG LONG Value in ILEC

I had been having problems with the declaration of LONG LONG values for testing in some recent code. The problem was compounded by the SEU/RSE interfaces not providing the same results.

The following code will compile in SEU but provides a compiler error in RSE!

#include
#include

main(){
unsigned long long tst1 = 18446744073709551615;

printf("tst1 = %llu\n",tst1);
}        

The error in RSE is

CZM0207 Integer constant 18446744073709551615 out of range.

However SEU managed to compile the code, but the output shows

tst1 = 4294967295

Obviously not what you would expect. Having posted the problem on the iSeriesNetwork forums I found out that the correct way to declare the variable is as follows

#include
#include

main(){
long long tst1 = 9223372036854775807LL;
unsigned long long tst2 = 18446744073709551615ULL

printf("tst1 = %ll\n",tst1);
printf(“tst2 = %llu\n”,tst2);
}                                

Now the code will compile on both compiler iterfaces and produce the correct results. I have asked IBM to update the manuals to show the correct way to declare a LONG LONG in a C program plus ensure the SEU and RSE interfaces are changed to be consistent in the output. I could have looked in the limits.h file and used the defined values (if I had half a brain!) which would have shown me the error of my ways! But if you do use LONG LONG variables in ILEC make sure you understand the correct method of initializing the variable with any value greater than the maximum allowed for a LONG!. postfixing LL or ULL could save a few hairs!

Chris…

Feb 28

Does Microsoft actually do anything with the reports it collects after problems?????

I have been having problems with Explorer not closing on a frequent basis, I have to go in and use task manager to force the end of the job! Everytime I do that I get the request, do I want to supply Microsoft with the information that is collected? I always say yes , they must have seen thousands of reports now? (Yes it crashes that often). My question is this, when I signed up to allow the collection of the data it was all about the Customer Satisfaction/Experience Program and how its important for Microsoft to collect this information to allow them to provide better solutions to their customer! How come after all of these reports is it still doing it? Is it really worth taking the time to follow the process or should I just ignore it?

I don’t see any customer experience improvement so far!

Chris…

Feb 28

Hacking attempts

I have been watching the logs for the webserver and see attempts from the following IP addresses to get into the site using scripts over the past 2 days.

89.241.145.134
213.37.240.40
83.43.228.109

They all originate from the following network according to whois!

OrgName: RIPE Network Coordination Centre
OrgID: RIPE
Address: P.O. Box 10096
City: Amsterdam
StateProv:
PostalCode: 1001EB
Country: NL

Looks like some script kiddy is trying to break into the site, for what reason I don’t know as no information is held on the site that isn’t already displayed to anyone who wants to browse round.

I have seen an increase over the past few months in this type of site scanning, maybe they like my site so much they are copying the content?

Anyhow looks like Amsterdam is a hot spot for this kind of activity at the moment!

Chris…

Feb 27

Sugar CRM and unsubscribe requests

We use Sugar CRM for our campaign management and email list management when we distribute information. It appears SugarCRM is ignoring the opt-out flag we have set it for a number of users? Firstly we apologize to those users who are still getting the emails from us even though they have requested to be removed, we are trying to fix the problem and have asked for support from Sugar to help us. We hope to have the problem fixed before our next email distribution and do endeavor to set the flag for every user that requested to be unsubscribed. If you do get further emails please be assured we are trying to fix the problem!

Chris…

Feb 27

Array Bounds checking ILEC Compiler

Looks like we have a problem Houston! I reported in an earlier post that we have been playing around with the QUILNGTX API and found a problem where the message was was overwriting the data stored in the UIM Panel on return. Looks like its a problem with the ILEC compiler and array bounds checking! IBM asked a very valid question how did we declare the variable being passed as the message, I reviewed the code and found we declared the character array with a maximum size of 255 bytes. This is just a legacy from previous message calls we made and didn’t strike a chord when we used the same declaration. We had the call to the popup window before the call to the QUIPUTV API which loads the panel variables initially but found that moving the call after the QUIPUTV call fixed up the issue we were seeing. So we felt that the problem was with the API and it was writing the data over the variable pool used for UIM. Wrong! Its is our building of the message that was creating the problem. We used a call to sprintf to build the first part of the message and multiple calls to the strcat function to add further data. No where did we get any error or warning messages even when we ran to programs. Our array is still overwriting memory it doesn’t own but we were fortunate (or not) that moving the API call masked the problem!

Looks like we need to ensure we do our own array bounds checking within every program as the compiler is not doing any which is what I thought it was? Here is the sample code which caused the problem.

#define MAX_MSG 255
char msg_dta[MAX_MSG];

if(Check_SMTP_Addr(RapRec.SMTPSRV) != 1) {
               /* set the error message and display again */
               Dsp_Err = 1;
               memcpy(smtpsts.srvaddr,RapRec.SMTPSRV,MAX_SMTP_NAME);
               smtpsts.flag = 0xf1;
               sprintf(msg_dta,"An attempt to contact the  SMTP server you ");
               strcat(msg_dta,"entered has failed.  Before the server can be");
               strcat(msg_dta,"configured the program must be able to ");
               strcat(msg_dta,"contact the server to verify its ");
               strcat(msg_dta,"authenticity. Please enter a valid SMTP ");
               strcat(msg_dta,"server address or *NONE ");
               strcat(msg_dta,"Here is some additional data which is not");
               strcat(msg_dta," part of the final message we will ship ");
               strcat(msg_dta,"but should show the problem????");
               Show_Popup_Error(msg_dta);
               QUIPUTV(applHandle,
                       &smtpsts,
                       sizeof(smtpsts),
                       "SMTPSTS   ",
                       &Error_Code);
               if(Error_Code.EC.Bytes_Available > 0) {
                  snd_error_msg(Error_Code);
                  exit(-1);
                  }

               }
            }

int Show_Popup_Error(char *message) {
int MsgLen = 0;
char Msg_Id[7] = "GEN0001";                 /* Message ID */
char Q_File_Name[20] = "RAPMSGF   *LIBL     "; /* Message file */
EC_t Error_Code = {'0'};                    /* Error Code struct */

Error_Code.EC.Bytes_Provided = sizeof(Error_Code);
MsgLen = strlen(message);

QUILNGTX(message,
         MsgLen,
         Msg_Id,
         Q_File_Name,
         &Error_Code);
if(Error_Code.EC.Bytes_Available > 0) {
   snd_error_msg(Error_Code);
   }
}  

As you can see I added some additional text to the message to show the problem occurring, I should have recognized the msg_dta array was being over filled but never thought about it!

So if you start to see some strange behavior when using arrays like this, understand that the ILEC compiler does not do array bounds checking by default (if at all?). I know other ILE compilers such as RPG do, so IBM may want to spend a little extra money on the C compiler and improve its capabilities (DIG). The latest code doesn’t use any of the techniques used in the above code so its not even a issue, this code was simply used to try out the call to the QUILNGTX API.

Chris…

Feb 27

QUILNGTX comes to the rescue

I have written numerous times now about the problems I have been having getting messages to be displayed to the right panel in UIM. I finally gave up trying to get the messages to show on the message line as no matter how I configured the QMHSNDPM API it would not place the message on the right panel once I started to nest panels. I then looked at the QUILNGTX API, it seemed to be capable of doing what I needed but did not appear to be UIM based (no application handle has to be sent to it!) so I was dubious about using it as an alternative. I had looked at simply using the QUIADDPW API and building the popups according to the type of panel I was using, but this seemed to be a simpler approach.

We started to code up the call to the API by passing in text which was built within the program using SPRINTF and STRCAT functions, the are some problems with this which IBM is looking into at the moment but overall it worked. The problem IBM is looking into is the placement of the call to the API in relation to other UIM calls which gave incorrect output, we moved the API calls around and fixed the problem temporarily and hope IBM can fix the problem permanently. The point is we can now have error messages sent to a popup window which is correctly placed over the right panel.

At first I was determined to get the messages to display at the bottom of the panel as this seemed to be the right approach and simply taking another route due to the difficulty of achieving that didn’t fit in with our mantra of trying to do it the right way. Having looked at the results we feel this is probably a better solution, if this was a GUI solution we would probably use a Dialog Box to display the message, so using a popup window is not unlike using that approach is it?

The code for the initial call to the API is below, we have altered this significantly to support replacement text and correct use of message files and allow the message to be formatted a little better.


int Show_Popup_Error(char *message) {
int MsgLen = 0;
char Msg_Id[7] = "GEN0001";                 /* Message ID */
char Q_File_Name[20] = "RAPMSGF   *LIBL     "; /* Message file */
EC_t Error_Code = {'0'};                    /* Error Code struct */

Error_Code.EC.Bytes_Provided = sizeof(Error_Code);
MsgLen = strlen(message);

QUILNGTX(message,
         MsgLen,
         Msg_Id,
         Q_File_Name,
         &Error_Code);
if(Error_Code.EC.Bytes_Available > 0) {
   snd_error_msg(Error_Code);
   }
} 

Note, the message passed in must be NULL terminated for this to work, The MsgID passed in will be used to determine the title of the popup window only. Again there appears to be a problem with that which we have asked IBM to comment on. The Message ID we passed in ‘GEN0001′ has no message text or help text so it did not show up the title problems.

I hope this helps others like us who have become frustrated with trying to get messages displayed at the bottom of display panels!

Chris…

Feb 27

Can you truely recover.

There are some companies out there that seem to think that just because they have implemented a HA solution everything is OK, if they have a failure they can simply switch to the remote system and they can continue as if nothing happened! If they are unfortunate enough to suffer a system loss during processing they could be in for a big shock!

Questions you need to ask yourself!

1. When was the last time I reviewed the configuration to make sure it was replicating everything I need?
2. When was the last time I did a planned switch over?
3. Can I recover the data to a known starting position from any process failure? If so how long will that take me?
4. If the system is running batch jobs can they be restarted if the system fails mid-way through?
5. Is the process keeping up with BOTH object and data replication consistently, if not how important is data to object synchronization?
6. If I get behind after my batch run how long before it catches up?

All of these questions are important and need to be understand before you start your recovery for the following reasons.

The systems we all run are changing on a constant basis, a new program added by a developer which is missed in the replication process could be fatal, you may not have the source available on the target to create the new object? Sod’s Law says its going to be the one you missed that stops it all from working! A planned switch will not prove you have a recoverable system in an unplanned event, its only going to confirm that you have the right objects and data being replicated to the target system. Should an unplanned event occur missing objects, down level objects and incomplete data are all possible.

If you have a batch process which produces lots of transactions for the database and it fails mid-way how do you know exactly what data needs to be removed, even if you know the data that needs to be removed the HA products provide no method of removing it! Object and data transports provided by most of the HA solutions do not work in unison, the data (journalled object data) is replicated automatically by the Remote Journal function as changes occur, objects tend to be replicated using a save and restore technology or command submission based around the audit journal identifying changes. This means the process of getting the objects to the remote system is always going to be delayed in comparison to journal entry based changes. The object save process and transmission to the remote system can have many bottlenecks even with the best solutions. I don’t think the object replication is as important as some would have you believe, but when implemented it does appear to be the major problem cause for replication backlogs and failures.

The level of journal and object replication backlogs will fluctuate through the day, at times it will be right up to date and other times be hours behind. Your challenge is knowing how to recover regardless of the backlog, remember data and objects follow different paths so they may have no consistency at all. One customer we worked with had a backlog every night of about 4 hours, the process normally caught up before the next batch ran but after an object failure got into a loop where it would never catch up! They had to do a full restore to the target just to get back to the normal backlog position.

HA is an ever moving target, what appears to function today may soon become obsolete and not provide the cover you need. The challenge is keeping one step ahead! Unless you have the skills to manage the configuration and regularly test its suitability by doing planned switches you may soon have a solution which is no longer serving the purpose for its existence. HA is not a finite art, all of us are still trying to deliver the perfect solution for the user and every time we get a little closer we find something else we need to consider.

I still hold onto my belief that while HA can be used as a DR solution, the fact that the environment and configuration works in a planned (HA) switch will not guarantee its success in a DR situation. You have to fully understanding the fragility of the data to application match up that the HA solutions provide to truely know your recoverability position. Having your data in a perfect synchronized position on both systems does not guarantee the application will restart should the source (production system) fail! The biggest problem most will have to work through is how do you know the application data is in such a state that no mismatch can occur? Remember Objects and journal data travels through different paths, so you will always have the opportunity for the data and objects to be out of sync! Add to this the fact that the majority of batch jobs which run out there have no recoverability built in and should a batch job be mid way when the failure occurs there could be significant work needed to recover. First have to know the batch was running then determine how to remove the data before restarting the batch job.

It may be easy to simply explain to the CEO or CFO that you have HA and you have recoverability, the only time you may be caught out is if you do have to recover your system during an unplanned event. Luckily for most the possibility of that happening is fairly remote. Or is it? We are not saying HA is not worthwhile, but we are saying look at what you have before you bury your head in the sand! For some HA is definitely overkill and simple data protection would be more than sufficient, others need something in between and therefore can justify HA as the only option available. Not setting expectations at the right level is going to catch you out in the long run!

Chris…

Feb 25

We have finally got into the early ship program for V6R1

For months now we have been trying to get an early release copy of V6R1 so we can test the products for running on V6R1. Finally we found the method of getting into the program and now have the DVD’s in transit for installation. The process was fairly easy to complete and the people in Rochester reacted quickly to get all of the documentation in place. The Documentation is already available on line for viewing on the online infocenter so we were able to review the Memo to users before we managed to get in touch with right people at IBM to get into the early release program, but the real test will be once we have the code to test with. If like us you have been trying to gain access but have not been able to send us a note via our website form and we will pass on the information.

A couple of really interesting areas we will look at are the new API’s and the journal changes which affect our RAP product and the way we can do more without having to change much in the way of product design and code. Luckily we have built the product to easily pick up any new features IBM adds to the journalling support without any changes required on our side.. The new IBM director interface may also help with adding new interfaces to the products as it does mention allowing user plug-ins to be created. I hope its a better offering than the old Ops Navigator capabilities as we found them to be severely lacking when it came to adding new functionality over the standard interface.

The first challenge will be the conversion process for the current product set and how we migrate to a new development environment which will support to code sets, we feel most companies would like to see a pre-converted copy of the objects available for download and not have to download, install and then wait for conversion to occur before being able to use the product no matter how quickly it occurs.

Once the new release is installed and we have tested the new features it promises to bring we will add more posts on our perceptions.

Chris…

Feb 21

Validate Email Address in C

We have been looking for ways to validate the email address (just make sure its correctly formed) and didn’t want to use the REGEXEC function so we thought we would try to find a method of doing it in straight C code against a passed string. We did find an example of how to do it for a ASCII based system but as the i5 is EBCDIC we had to change the program somewhat to make it work. We have not passed every type of email address through because there are so many, but using a set of addresses from our address book all passed and where we changed them to a format which should fail they did!

We have a EBCDIC to ASCII translator built which came in very handy as the string has to be parsed to ASCII to allow the allowed characters to be checked for (ASCII 32 – 127). We did look at simply converting the whole code to run against EBCDIC but the spread of the characters in numeric terms didn’t make that look easy! We could have converted each character back to EBCDIC as we needed to but this was a simpler route (yeah we sometimes take the easy route).

There are plenty of Conversion programs around so we don’t show it here. The rest of the code should run OK

int Check_Email_Addr(char *EM_Addr) {
int count = 0;
int i = 0;
char conv_buf[MAX_EMAIL_NAME];
char *c, *domain;
char *special_chars = "()<>@,;:\\\"[]";

/* The input is in EBCDIC so convert to ASCII first */
strcpy(conv_buf,EM_Addr);
EtoA(conv_buf);
/* convert the special chars to ASCII */
EtoA(special_chars);

for(c = conv_buf; *c; c++) {
   /* if '"' and beginning or previous is a '.' or '"' */
   if (*c == 34 && (c == conv_buf || *(c - 1) == 46 || *(c - 1) == 34)) {
      while (*++c) {
         /* if '"' break, End of name */
         if (*c == 34)
            break;
         /* if '\' and ' ' */  
         if (*c == 92 && (*++c == 32))
            continue;
         /* if not between ' ' & '~' */   
         if (*c <= 32 || *c > 127)
            return 0;
         }
      /* if no more characters error */   
      if (!*c++)
         return 0;
      /* found '@' */   
      if (*c == 64)
         break;
      /* '.' required */   
      if (*c != 46)
         return 0;
      continue;
      }
   if (*c == 64) {
      break;
      }
   /* make sure between ' ' && '~' */
   if (*c <= 32 || *c > 127) {
      return 0;
      }
   /* check special chars */   
   if (strchr(special_chars, *c)) {
      return 0;
      }
   } /* end of for loop */
/* found '@' */   
/* if at beginning or previous = '.' */
if (c == conv_buf || *(c - 1) == 46) 
   return 0;
/* next we validate the domain portion */
/* if the next character is NULL */
/* need domain ! */
if (!*(domain = ++c))
   return 0;
do {
   /* if '.' */
   if (*c == 46) {
      /* if beginning or previous = '.' */
      if (c == domain || *(c - 1) == 46)
         return 0;
      /* count '.' need at least 1 */   
      count++;
      }
   /* make sure between ' ' and '~' */   
   if (*c <= 32 || *c >= 127)
      return 0;
   if (strchr(special_chars, *c))
      return 0;
   } while (*++c); /* while valid char */
return (count >= 1); /* return true if more than 1 '.' */
}    

We have put lots of comments into the code to show how it works. The original idea came from a book called Secure Programming Cookbook from O’Reilly.
The biggest problem was converting the character set but taking the option to convert it all to ASCII first and then map the ASCII characters for the checking works well. We will review the REGEXEC method later to see if it is a better alternative but at this time we do have a working function.

Chris…