Nov 30

Solution for the QDBRPLAY API offset issues

To those who have been following the saga, we have finally come up with what we feel is a suitable answer to the problem with the Journal Entry changes made by IBM when they moved from V5R4 to V6R1. the changes we have implemented will be updated once IBM comes back with a new structure layout for the headers.

First we had to change the header for the ESD_Template so that it would allow us to reference the new offset variable.

typedef _Packed struct  My_ESD_Struct_x {
                        char DbJrnl_File_Name[10];
                        char DbJrnl_File_Lib[10];
                        char DbJrnl_Member_Name[10];
                        char DbJrnl_New_File_Name[10];
                        char DbJrnl_New_File_Lib[10];
                        char DbJrnl_New_Member_Name[10];
                        char DbJrnl_Reserved1[8];
                        int  offset_to_ptr;
                        int  DbJrnl_Apply_Length;
                        char DbJrnl_Reserved2[32];
                        char DbJrnl_Subtype;
                        char DbJrnl_Reserved3[3];
                        short DbJrnl_Chg_Trg_Library_Len;
                        int  DbJrnl_Chg_Trg_Library_Offset;
                        short DbJrnl_Chg_Trg_Name_Len;
                        int  DbJrnl_Chg_Trg_Name_Offset;
                        char DbJrnl_Attr[10];
                        short DbJrnl_Number_Based;
                        int DbJrnl_Based_Names_Offset;
                        short DbJrnl_Based_Names_Next_Offset;
                        char DbJrnl_Not_Logged;
                        char DbJrnl_Reserved4[1];
                        }My_ESD_Struct_t;   

The change just creates a new (int)variable offset_to_ptr which resides in the last 4 bytes of the Reserved1 character field. The pointer is stored at the end of this structure in a V5R4 entry but has moved 16 bytes in a V6R1 entry so our original structure definition worked for V5R4 entries but not V6R1 entries.
Note:
The Journal entry is changed only by OS Version it was deposited on, an entry which is deposited on a V5R4 system will replay correctly on a V5R4 and V6R1 system using the headers IBM supplied. A V6R1 entry would not replay on either a V6R1 system or a V5R4 system using those same headers.

The next structure simulates the character pointer we used in the original structure which was built from the IBM structure plus a couple of extra character pointers attached on the end.

typedef _Packed struct  My_ESD_Ptr_x {
                        char *esd_data;
                        }My_ESD_Ptr_t; 

To move the above structure to the correct position we used the following code

char *tmp2;
My_ESD_Struct_t *My_ESD_Template;
My_ESD_Ptr_t * My_ESD_Data; 

My_ESD_Template = (My_ESD_Struct_t *)&Data->ESD;
tmp2 = &Data->ESD;
tmp2 += My_ESD_Template->offset_to_ptr;
My_ESD_Data = (My_ESD_Ptr_t *)tmp2;

Now a memcpy of the data from My_ESD_Data->esd_data brings in the correct data to the API.

The Headers may get updated by IBM as may the manuals. One key piece of information which may be significant is the pointer is always sat in the last 16 bytes of the ESD header contained in the journal entry. The only reason this became a problem is IBM changed the length of the ESD header without mentioning it in any correspondence, so the header defined in QSYSINC/H was 16 bytes shorter than the actual data.

I hope that clarifies the situation and how to code round the incorrect information in the manuals and headers.

Chris..

Nov 25

Still have people who want something for free but wont say who they are.

One thing which really gets to me is where you provide a service to people and they seem to need to lie about who they are. In this particular case the IP address was logged and came from an IBM Premier Business Partner in Germany. That’s not to say it wasn’t relayed via this IP, but looking at the information and what they downloaded 5733SC1 for V5R4 I would say it probably wasn’t.

The point I am making is this, I provide the service free of charge to those who decide to register with the site. I don’t generally use the information collected for spamming the registered users, I simply follow up if they have downloaded one of our products. If this did come from the IBM BP in question they certainly have the same access rights into IBM as we do, so why steal from us! Strong words I know, but I really get miffed at these jokers. I don’t see any downloads available from their site yet they feel its OK to go to others who bother to provide a service and take without following the rules!

We bother to help the community and give back in the services we offer and the information we pass out. The least users can do is be truthful and say who they are, removing the service affects everyone not just the offenders! This costs us money to manage and provide the services so please, if you need something at least have the decency to be truthful with us.

Chris…

Nov 24

Maybe not that stupid after all! QDBRPLAY temporary solution (not production)

I am still trying to get IBM to provide a production level solution for this problem, but in the in the meantime I have been able to take the information supplied from IBM and code around the issues to allow the programs to correctly apply Journal Entries generated on both V6R1 & V5R4 systems.

The solution is not production ready because it requires assumptions to be made that in a real world may not be correct (IBM is still sending me confusing information about the offset placement). The biggest being the state of the ESD pointer, I am assuming if it is NULL then the journal entry was probably created on a V6R1 system and I should use the IBM offsets to get to the data required for the QDBRPLAY API. Otherwise use the pointer I had used before…

This is the test code I originally developed for IBM to show the issue, I am not building all of the data for the QDBRPLAY API just making sure it is available. You will need to add a lot of code to get the journal entries and start the process but this is about the QDBRPLAY API issue where the pointer to additional data is incorrectly positioned in the structure. But if you understand this part of the exercise you can use it to complete your journal entry apply process.

   jrne = space;
   jrne += Jrn_Spc_Hdr->Offset_First_Jrn_Entry;
   Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
   Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
   printf("Num Ents = %d\n",Num_Ents);
   printf("Size of QSYSINC Structure = %d\n",
          sizeof(_Packed struct QdbJrnl_DDL0100));
   for(i = 0; i < Num_Ents; i++) {

      printf("Type = %.2s\n",Jrn_Spc_Ent->Entry_Type);
      printf("Code = %c\n",Jrn_Spc_Ent->Jrn_Code);
      printf("Seq_num found %lld\n",Entry_Seq_Number);
      tmp = jrne;
      /* need to make sure the incomplete data bit is set */
      /* Jrn_Spc_Ent->Incomplete_Data   */
      tmp += Jrn_Spc_Ent->Dsp_Entry_Specific_Data;
      Data = (Qjo_RJNE0100_JE_ESD_t *)tmp;
      ESD_Len = QXXZTOI(Data->Len_ESD,5,0);
      printf("ESD Len = %d\n",ESD_Len);
      memcpy(Input_Template.Base.Journal_Type,
             Jrn_Spc_Ent->Entry_Type,2);
      Input_Template.Base.Journal_Code = Jrn_Spc_Ent->Jrn_Code;
      /* move esddta to the data in case no additional data */
      printf("Referential Constraint %d\n",Jrn_Spc_Ent->Referential_Constraint);
      printf("Trigger %d\n",Jrn_Spc_Ent->Trigger);
      printf("Incomplete data %d\n",Jrn_Spc_Ent->Incomplete_Data);
      printf("Ignore during APYRMV %d\n",Jrn_Spc_Ent->Ignore_during_APYRMV);
      printf("Min ESD %d\n",Jrn_Spc_Ent->Min_ESD);
      printf("File Type Indicator %d\n",Jrn_Spc_Ent->File_Type_Indicator);
      printf("Min Field Boundaries %d\n",Jrn_Spc_Ent->Min_Field_Boundaries);
      printf("Reserved1 %d\n",Jrn_Spc_Ent->Reserved1);
      if(Jrn_Spc_Ent->Incomplete_Data == 1) {
         if(Crt_Usr_Spc(ESD_Dta_SPC,_16MB) != 1) {
            printf("Could not create ESD Space\n");
            return -1;
            }
         QUSPTRUS(ESD_Dta_SPC,
                  &esddta,
                  &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Could not get pointer to ESD Space\n");
            QUSDLTUS(Ent_Req_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               Error_Code.EC.Bytes_Available = 0;
               }
            QUSDLTUS(Jrn_Ent_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               }
            return -1;
            }
         /* set the template for the ESD */
         ESD_Template = (ESD_Tmp_Str_t *)&Data->ESD;
         /* move ptr to UsrSpace for storing data */
         tmp1 = esddta;
         /* move the data to the usrspc from the journal entry */
         memcpy(tmp1,&Data->ESD,(ESD_Len - 16));
         tmp1 += (ESD_Len - 16);
         /* pointer error when hign mem usage */
         printf("ESD Data Length TS %d\n",
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         memcpy(tmp1,ESD_Template->Int_Data,
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");
      /* remove the ESD pointer */
      if(Jrn_Spc_Ent->Ptr_Handle != 0) {
         QjoDeletePointerHandle(&Jrn_Spc_Ent->Ptr_Handle,
                                &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Failed to delete pointer handle\n");
            }
         QUSDLTUS(ESD_Dta_SPC,
                  &Error_Code);
         }
      if(Jrn_Spc_Ent->Dsp_Next_Jrn_Hdr > 0) {
         jrne += Jrn_Spc_Ent->Dsp_Next_Jrn_Hdr;
         Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
         Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
         }
      Start_Seq_Number = Entry_Seq_Number;
      Start_Seq_Number ++;
      }
   } /* else */ 

As explained in the previous post this works fine when applied against a journal receiver created on a V5R4 system. But fell down when trying to map over a receivers entries generated in V6R1. SO I have added the following code in the appropriate place

   if(ESD_Template->Int_Data != NULL) {
      /* if its not NULL I can use it, probably from V5R4?? */ 
            memcpy(tmp1,ESD_Template->Int_Data,
                   ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
            }
         else {
            /* I assume its a V6R1 entry */
            esd_offset = tmp;
            esd_offset += 0x44;
            memcpy(tmp1,esd_offset,ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
            }
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");

Now, while the above code works it is using totally undocumented information so I would say its probably able to change?

If you take the above offset and add it to the start of the ESD information it will actually map to the last 4 bytes of – char DbJrnl_Reserved1[12]; – in the QdbJrnl_DDL0100 structure, I assume that IBM will probably amend the header file to open up that information to help resolve the problem, but it is flawed when addressing a V5R4 entry because its not set in those entries for some reason?

As more information becomes available I will continue to post it here and in the forums for those of you who are affected by this problem.

Chris…

Nov 21

I am REALLY that STUPID???

Gentlemen this is a long Post, please make yourself comfortable. I am only posting this because I am totally confused by IBM’s response and fully accept that I am totally stupid and should get a better day job. I have been trying to get to the bottom of this since Tuesday after raising a SEV 1 error with IBM. The course it took is a story on its own, I will share that with those who are interested separately (Dirty Washing)

If you already read the iSeriesNetwork forums, you can skip this one! Its an exact copy of the one I posted in the forums there with just a few spelling errors changed and a couple of presentation enhancements.

The problem is in the journalling area and is based on what I have managed to glean from the manuals supplied by IBM, I have to admit that being English has not helped with the interpretation of the content of the manuals so I may have totally misunderstand what is being said. For those who don’t have English as your first language you may totally miss the interpretation, sorry for that!

OK so first of all lets look at what information I started to work with

Here is the starting place, it is from the Manuals about the use of the QjoREtrieveJournalEntries. This is the part of the structure definitions which describe the RJNE0200_JE_Hdr_t. I have cut down the data for expedience as the rest doesn’t affect the problem I am having.

218(0) DA(0) BIT(1) Referential constraint
218(1) DA(1) BIT(1) Trigger
218(2) DA(2) BIT(1) Incomplete data
218(3) DA(3) BIT(1) Ignored during APYJRNCHG or RMVJRNCHG
218(4) DA(4) BIT(1) Minimized entry specific data
218(5) DA(5) BIT(1) File type indicator
218(6) DA(6) BIT(1) Minimized on field boundaries
218(7) DA(7) BIT(1) Reserved

I had raised a question previously in the forums about how to manipulate the bit fields You can find that post on the iSeriesNetwork forums.

Here is the Output from QSYSINC/H(QJOURNAL) Qjo_RJNE0200_JE_Hdr_t which matches the information above

unsigned int Referential_Constraint:1; /* @ACA @
unsigned int Trigger:1; /* @ACA @
unsigned int Incomplete_Data:1; /* @ACA @
unsigned int Ignore_during_APYRMV:1; /* @ACA @
unsigned int Min_ESD:1; /* @ACA @
unsigned int File_Type_Indicator:1; /* @
unsigned int Min_Field_Boundaries:1; /* @
unsigned int Reserved1:1; /* @ACA @AGC @AKA @

So while the manual states the flags are bits, the C structure states they are Unsigned Int? But that really doesn’t matter either because the C program using the above structure from the QSYSINC/H file returns the correct information.

Here is the quote from the manual about the Incomplete data flag

Incomplete data. Whether this entry has data that must be additionally retrieved using a pointer returned for the missing information. See Use of Pointers within Entry Specific Data for more information. The possible values are:
0 This entry does not have any pointers included.
1 This entry does have pointers included.

The pointer within the text sends the reader to the following information, I left it more or less intact but you could read it online for the full version if needed. I also said this post would be long…

Use of Pointers within Entry Specific Data

There are some journal entries that require additional handling of the journal receiver entry specific data using pointers. This was done to minimize movement of large amounts of data and to facilitate support of tables or database files with large object (LOB) fields. The types of entries that may require pointer support are:

* Any operations on specific records or files (journal code R or F) of tables or database files that include any fields of data type BLOB (binary large object), CLOB (character large object), or DBCLOB (double-byte character large object). See the DB2 UDB for iSeries SQL Programming Concepts and DB2 UDB for iSeries SQL Reference books for more information on these data types.
* Operations related to byte stream file write operations, Journal Code B, Entry type WA. See the Integrated file system information for more information about these journal entries.
* Operations related to data queue send operations, Journal Code Q, Entry types QK and QS. See the Journal management topic for more information on these journal entries.
* Any operations on specific records or files (journal code R or F) of tables or database files resulting in minimized entry specific data when the journal has MINENTDTA specified for the corresponding object type. See the Journal management topic for restrictions and usage of journal entries with minimized entry specific data.

If the incomplete data indicator is returned as a 1, then that indicates that the journal-entry specific data includes a pointer to additional data. Additionally, a pointer handle will be returned with the journal entry. This handle is associated with any allocations required to support the pointer processing.

The pointer must be used by the same process that called this API; it cannot be stored and used by a different process. The pointer can be used for read access only. See the Journal management topic for descriptions of the entry types that may include pointer data. The pointer can be used in the following way:

* It can be used directly to copy the data addressed to some other storage space.
* If the journal entry is a record entry (journal code R), the journal-entry specific data could be used for an update or insert operation to the database file through SQL. See the DB2 UDB for iSeries SQL Reference book for more information.

The pointer handles will be implicitly deleted when the process that requested the journal entries is ended.

These pointers can be used only with the V4R4M0 or later versions of the following languages:

* ILE COBOL
* ILE RPG
* ILE C if the TERASPACE parameter is used when compiling the program. See the WebSphere Development Studio: ILE C/C++ Programmer’s GuideLink to PDF book for more information.

Once the pointer data is used, you must delete the pointer handle to free the handle and any allocations associated with that handle. This can be done by using the Delete Pointer Handle (QjoDeletePointerHandle) API. If the handles are not deleted, the maximum number allowed can be reached, which will prevent further retrieval of journal entries. The deletion must occur from the same process that called the Retrieve Journal Entries (QjoRetrieveJournalEntries) API.

Even if the journal entry data is not used, all pointer handles returned to the user through this interface should be deleted. This is also true when partial journal entry information is returned, even though an error message was returned.

I have included the part of the program which maps the relevant structures over the data, remember these are the same regardless of what OS level the Journal entry I am mapping over comes from.(this point becomes clearer later I hope?)


   jrne = space;
   jrne += Jrn_Spc_Hdr->Offset_First_Jrn_Entry;
   Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
   Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
   printf("Num Ents = %d\n",Num_Ents);
   for(i = 0; i < Num_Ents; i++) {
      printf("Type = %.2s\n",Jrn_Spc_Ent->Entry_Type);
      printf("Code = %c\n",Jrn_Spc_Ent->Jrn_Code);
      printf("Seq_num found %lld\n",Entry_Seq_Number);
      tmp = jrne;
      /* need to make sure the incomplete data bit is set */
      /* Jrn_Spc_Ent->Incomplete_Data   */
      tmp += Jrn_Spc_Ent->Dsp_Entry_Specific_Data;
      Data = (Qjo_RJNE0100_JE_ESD_t *)tmp;
      ESD_Len = QXXZTOI(Data->Len_ESD,5,0);
      printf("ESD Len = %d\n",ESD_Len);
      memcpy(Input_Template.Base.Journal_Type,
             Jrn_Spc_Ent->Entry_Type,2);
      Input_Template.Base.Journal_Code = Jrn_Spc_Ent->Jrn_Code;
      /* move esddta to the data in case no additional data */
      printf("Referential Constraint %d\n",Jrn_Spc_Ent->Referential_Constraint);
      printf("Trigger %d\n",Jrn_Spc_Ent->Trigger);
      printf("Incomplete data %d\n",Jrn_Spc_Ent->Incomplete_Data);
      printf("Ignore during APYRMV %d\n",Jrn_Spc_Ent->Ignore_during_APYRMV);
      printf("Min ESD %d\n",Jrn_Spc_Ent->Min_ESD);
      printf("File Type Indicator %d\n",Jrn_Spc_Ent->File_Type_Indicator);
      printf("Min Field Boundaries %d\n",Jrn_Spc_Ent->Min_Field_Boundaries);
      printf("Reserved1 %d\n",Jrn_Spc_Ent->Reserved1);
      if(Jrn_Spc_Ent->Incomplete_Data == 1) {
         if(Crt_Usr_Spc(ESD_Dta_SPC,_16MB) != 1) {
            printf("Could not create ESD Space\n");
            return -1;
            }
         QUSPTRUS(ESD_Dta_SPC,
                  &esddta,
                  &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Could not get pointer to ESD Space\n");
            QUSDLTUS(Ent_Req_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               Error_Code.EC.Bytes_Available = 0;
               }
            QUSDLTUS(Jrn_Ent_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               }
            return -1;
            }
         /* set the template for the ESD */
         ESD_Template = (ESD_Tmp_Str_t *)&Data->ESD;
         /* move ptr to UsrSpace for storing data */
         tmp1 = esddta;
         /* move the data to the usrspc from the journal entry */
         memcpy(tmp1,&Data->ESD,(ESD_Len - 16));
         tmp1 += (ESD_Len - 16);
         /* pointer error when hign mem usage */
         printf("ESD Data Length TS %d\n",
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         memcpy(tmp1,ESD_Template->Int_Data,
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");

Couple of points to make, firstly I am using the IBM supplied structures where possible and simply adding the relevant additions such as a Character pointer to the structure which maps the ESD_Data. I could post all of the information but feel it is irrelevant due to the fact that the maps you will see later actually return sensible data in most cases. Secondly if the structures were wrong I would expect other pointer errors to be returned as I move the data around.

So now down to the data I am working with. I had a response from IBM as follows

Hi Chris,
I did not originally work on this issue but tought it would be better if Peter spoke with you about
devellopements response, I will forward it to you:
Developement ran the recreation scenario and can see the D CT entry
on our system. DSPJRN indicates Incomplete data .. No, Therefore there
is no ptr in this jrn entry. Looks like our customer’s code is
expecting there to be one. If the entry is not >32KB you don’t
get a ptr, as documented in infocenter for the D CT entry:
.
Notes:
1. This data does not apply to integrated file system objects.
2. If the data for these entries exceeds 32 KB, then a pointer is
returned to the actual data when the entry is retrieved using an
option to return pointers. If the return pointer option is not
used, then *POINTER is returned for the entry-specific data.
So this not a system code bug, the issue is with the application code.
thanks
Robert

Thank you for using IBM products and support.

So IBM did their debug by taking the journal receiver and journal I had sent and did a DSPJRN against it? I was amazed! I am not using DSPJRN I am using an API? They have the code and they have the object, they insisted on all of that before they would even consider looking at it. Why if they only ran a DSPJRN to come up with their answer (Its the stupid user?) did they need it?

Then

Hi Chris,

Thanks for the emails. I’ve spoken to your local support and they will address your questions on Monday.

Regards, Debbie

IBM Australia

I won’t post my response but to say I am not happy after having worked non stop on trying to find out where I am making my stupid assumptions and why IBM must be right is not going to get past the rules of decency… Why they would not talk to me direct is also comical…

So having been told I am an idiot and I have wait until Monday to be told why, I decided to review all of the information I have found and shared with IBM. This post is the last part of this until Monday unless someone can help me see the error of my ways.

First of all you will see IBM is dismissing my problem as a User problem because DSPJRN shows the Incomplete Data Flag is set to No? This made me think Oh my God what an Idiot! Why did I make such a stupid mistake! So I went back through my code to see why..

Based on my reading of the manual data I assumed that if the Incomplete Data Flag is set there should be a Pointer in the returned Journal Entry which is pointing to the data to be retrieved, that pointer should therefore not be NULL? I mean whats the point in saying yes there is a pointer in this data but it may be NULL?? Did I make the biggest fundamental mistake there?

So I went through the scenario’s AGAIN. Remember (its been a long story so far..) that this only seems to occur when the journal entry I am working with is generated on a V6R1 system, a journal entry for the exact same process on a V5R4 system actually works OK. All I am doing as can be seen from the code above is trying to create a file from the data contained in a journal entry The process created a D CT entry which I am using the QDBRPLAY API to replay…

This information is from the system running V5R4. The data in the journal entry is therefore created on a V5R4 system (clear?)

First I looked for an entry which would describe my problem and displayed the information in the same manner as Rochester appears to have. I found an entry in TST0000002 which is attached to #tstjrn/tstjrn.

Here is the output

Display Journal Entries

Journal . . . . . . : TSTJRN Library . . . . . . : #TSTJRN
Largest sequence number on this screen . . . . . . : 00000000000000000013
Type options, press Enter.
5=Display entire entry

Opt Sequence Code Type Object Library Job Time
3 J PR CHRISH#VIB 14:25:33
4 J JR TST0000002 #TSTRCV CHRISH#VIB 14:25:33
5 I UE #DTA QSYS CHRISH#VIB 14:26:39
6 J XP F #DTA CHRISH#VIB 14:26:56
7 I UE #DTA QSYS CHRISH#VIB 14:26:56
8 D CT F #DTA CHRISH#VIB 14:26:56
9 D JF F #DTA CHRISH#VIB 14:26:56
10 F JM F #DTA CHRISH#VIB 14:26:56
11 D MA F #DTA CHRISH#VIB 14:26:56
12 F MC F #DTA CHRISH#VIB 14:26:56
13 J NR CHRISH#VIB 14:27:23

Bottom
F3=Exit F12=Cancel

Display the specific journal entry for the CT

Display Journal Entry

Object . . . . . . . : F Library . . . . . . : #DTA
Member . . . . . . . :
Incomplete data . . : No Minimized entry data : *NONE
Sequence . . . . . . : 8
Code . . . . . . . . : D – Database file operation
Type . . . . . . . . : CT – Create database file

Entry specific data
Column *…+….1….+….2….+….3….+….4….+….5
00001 ‘F #DTA ‘
00051 ‘ µ Ø  Z 1091117′
00101 ‘142656 9 PF 00°Ø ¨³’
00151 ‘¹ {Y N *CHANGE ‘
00201 ‘ 1091117142656 ‘
00251 ‘ ‘
00301 ‘ DN ¯ Y ‘
More…
Press Enter to continue.

F3=Exit F6=Display only entry specific data
F10=Display only entry details F12=Cancel F24=More keys

**Please note that the Incomplete data flag is set to No on this output.

Here is the output of the program above

Num Ents = 2
Type = CT
Code = D
Seq_num found 8 <<<<<<<<<<<<< ESD Len = 176
Referential Constraint 0
Trigger 0
Incomplete data 1 <<<<<< Note!!!
Ignore during APYRMV 0
Min ESD 0
File Type Indicator 0
Min Field Boundaries 0
Reserved1 0
ESD Data Length TS 1664
Press ENTER to end terminal session.

So while the DSPJRN entry display is saying the Incomplete Data flag is set to No my program seems to say the flag contained in the structure within the actual journal entry is saying it is?

Note: The program completes without error, but just in case I was wrong I thought I would debug the structures to see if the flag is just throwing me a curve ball?

Here is the debug data from the ESD_Template structure, notice the pointer is actually set?

EVAL *ESD_Template
(*ESD_Template).ESD_Hdr.DbJrnl_File_Name = SPP:38793ED09E001160
(*ESD_Template).ESD_Hdr.DbJrnl_File_Lib = SPP:38793ED09E00116A
(*ESD_Template).ESD_Hdr.DbJrnl_Member_Name = SPP:38793ED09E001174
(*ESD_Template).ESD_Hdr.DbJrnl_New_File_Name = SPP:38793ED09E00117E
(*ESD_Template).ESD_Hdr.DbJrnl_New_File_Lib = SPP:38793ED09E001188
(*ESD_Template).ESD_Hdr.DbJrnl_New_Member_Name = SPP:38793ED09E001192
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved1 = SPP:38793ED09E00119C
(*ESD_Template).ESD_Hdr.DbJrnl_Apply_Length = 1648
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved2 = SPP:38793ED09E0011AC
(*ESD_Template).ESD_Hdr.DbJrnl_Subtype = ‘9’
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved3 = SPP:38793ED09E0011CD
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Library_Len = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Library_Off set = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Name_Len = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Name_Offset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Attr = SPP:38793ED09E0011DC
(*ESD_Template).ESD_Hdr.DbJrnl_Number_Based = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Based_Names_Offset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Based_Names_Next_Of fset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Not_Logged = ‘0’
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved4 = SPP:38793ED09E0011EF
(*ESD_Template).Int_Data = SPP:00008000080E64EE <<< Note!!!

So this seems to say the program is working as I expected it to? ( I have already admitted to be stupid, but maybe not so?)

I then thought OK I will do the same for a journal entry on the V6R1 system, maybe I have just mis-understood what I thought I was seeing. After all I was copying the receiver from a remote journal and then attaching it to the local journal before running the program against it. Perhaps that’s why the pointer problems APPEAR to exist?

Here is the DSPJRN against an entry created using the exact same process on the V5R4 system to the V6R1 system eg: CRTPF FILE(#DTA/G) RCDLEN(100)

Display Journal Entry

Object . . . . . . . : G Library . . . . . . : #DTA
Member . . . . . . . :
Incomplete data . . : No Minimized entry data : *NONE
Sequence . . . . . . : 133
Code . . . . . . . . : D – Database file operation
Type . . . . . . . . : CT – Create database file

Entry specific data
Column *…+….1….+….2….+….3….+….4….+….5
00001 ‘G #DTA ‘
00051 ‘ µ Ø  Z 1091121′
00101 ‘095550 9 PF 00°Ø ¨³’
00151 ‘¹ {Y N *CHANGE ‘
00201 ‘ 1091121095550 ‘
00251 ‘ ‘
00301 ‘ DN ¯ Y ‘
More…
Press Enter to continue.

F3=Exit F6=Display only entry specific data
F10=Display only entry details F12=Cancel F24=More keys

The Incomplete Data flag is still No? OK so that’s not changed? So I will run the program in debug as usual, here is the output.

EVAL *ESD_Template
(*ESD_Template).ESD_Hdr.DbJrnl_File_Name = SPP:1F9BB40FCB001160
(*ESD_Template).ESD_Hdr.DbJrnl_File_Lib = SPP:1F9BB40FCB00116A
(*ESD_Template).ESD_Hdr.DbJrnl_Member_Name = SPP:1F9BB40FCB001174
(*ESD_Template).ESD_Hdr.DbJrnl_New_File_Name = SPP:1F9BB40FCB00117E
(*ESD_Template).ESD_Hdr.DbJrnl_New_File_Lib = SPP:1F9BB40FCB001188
(*ESD_Template).ESD_Hdr.DbJrnl_New_Member_Name = SPP:1F9BB40FCB001192
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved1 = SPP:1F9BB40FCB00119C
(*ESD_Template).ESD_Hdr.DbJrnl_Apply_Length = 1664
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved2 = SPP:1F9BB40FCB0011AC
(*ESD_Template).ESD_Hdr.DbJrnl_Subtype = ‘9’
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved3 = SPP:1F9BB40FCB0011CD
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Library_Len = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Library_Off set = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Name_Len = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Chg_Trg_Name_Offset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Attr = SPP:1F9BB40FCB0011DC
(*ESD_Template).ESD_Hdr.DbJrnl_Number_Based = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Based_Names_Offset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Based_Names_Next_Of fset = 0
(*ESD_Template).ESD_Hdr.DbJrnl_Not_Logged = ‘0’
(*ESD_Template).ESD_Hdr.DbJrnl_Reserved4 = SPP:1F9BB40FCB0011EF
(*ESD_Template).Int_Data = SPP:*NULL

Sure enough the pointer is NULL, but what about the check for the Incomplete flag in the program why did it skiip past that, is the logic wrong?

Here is the output from the program

Num Ents = 2
Type = CT
Code = D
Seq_num found 133
ESD Len = 176
Referential Constraint 0
Trigger 0
Incomplete data 1 <<<< Hmmm it appears to be set...
Ignore during APYRMV 0
Min ESD 0
File Type Indicator 0
Min Field Boundaries 0
Reserved1 0
ESD Data Length TS 1664

But this time as you would expect based on the data shown via the debug screens the pointer I am trying to move data from is NULL…. So the program actually ends abnormally with MCH3601 error….

IBM did say this was probably a programming error and it is, I am trying to copy data from a NULL pointer, but I am only doing that because the information supplied by the IBM API is telling me I should… isn’t it???

By the way, if you have got this far you deserve a medal!

I can obviously code around this slight problem by checking the pointer is not NULL. But the question I have is, should I do this? I only try to move the data because IBM is telling me via their returned data that I should do it?

Is the fact that the DSPJRN view is wrong as well, not an issue? It says there is no pointer to data even when there is (V5R4 v V6R1)

Do we not have a practice of providing backwards compatibility? If the programs works perfectly on V5R4 I assumed it would work just as perfectly on V6R1? Yes I scoured the Memo to Users to see if I needed to do anything different, the only problem I found was the User Space size had been reduced from a true 16MB to only support 16773120 bytes.. I had done that on the initial release.

While this seems to answer itself to me, I need more eyes over it to see what assumptions that I make are flawed. I am after all just a stupid programmer that has difficulty reading manuals… If the English in the post is bad, sorry the manuals have done that to me also…

I would appreciate any feedback you have, no the program is not production code, its just to show the problem… Have I made too many assumptions..? Even if you think I have made the right assumptions your replies as such would at least help raise my esteem a little… No Rocks to hide under I am afraid and this has to be sorted for my sanity if nothing else…

Many thanks for even taking the time to read this rant…..

Chris…

Nov 17

How to remove invalid IFS names

We had a client call after he had managed to create an IFS directory which could not be deleted using the WRKLNK interface or the RMDIR command using our new FTP Client. We didn’t realize that the IBM OS based commands could not delete items on the IFS under certain circumstances, but we also found that those same commands would refuse to create the objects as well.

The client had created a directory called c:\ftpclnt in the root directory, trying to delete the item using option 4 from the WRKLNK command failed as did the RMDIR command with a message stating the object could not be found.

Here is a sample of the message sent

Additional Message Information

Message ID . . . . . . : CPFA0A9 Severity . . . . . . . : 40
Message type . . . . . : Diagnostic
Date sent . . . . . . : 11/17/09 Time sent . . . . . . : 18:09:12

Message . . . . : Object not found. Object is /c:/testdir.
Cause . . . . . : Object /c:/testdir, or a directory in the object path,
could not be found, or its type cannot be resolved by this function.
Recovery . . . : Correct the name or specify an object of the correct type.
To determine if the object exists, use the Work with Object Links (WRKLNK)
command. If the name exists, check the type of the object. If the name
contains symbolic link objects, ensure the path names they resolve to exist.
Retry the operation.

Bottom
Press Enter to continue.

F3=Exit F6=Print F9=Display message details
F10=Display messages in job log F12=Cancel F21=Select assistance level

Apparently this is because the OS commands interpret the \ character differently,as can be see above it created a path of /c:/testdir which reversed the ‘\’ and called it a directory yet didn’t create a subdirectory of /testdir under c:….
We looked at the code which allowed the object to be created and it was the Unix API mkdir(). We also tested the delete of the link using our product and it did as we expected and successfully deleted the object. The question was why did the IBM commands refuse to find the object.

We logged a call with IBM support and they sent us to the following information . Talking with support we explained that we had Googled the problem and didn’t find the document they had sent, they explained this was because the document is contained in a knowledge base that is not viewable by web robots.

So if you are having the same problems, hopefully Google will find this reference and save you some time.

I have also copied the content below just in case the link doesn’t work for some? The customer said his link was only able to be deleted by using Navigator or our program, so make sure you try a few options.

Document Title: Renaming or Removing Files from the Integrated File System That Have Names That Are Not Valid

Abstract

This document provides tips on how to delete documents and directories in the Integrated File System that were created with invalid names.

Document Description:
Certain applications may create invalid file names in the operating system Integrated File System. For example, a file name containing slashes or quotes is not valid and cannot be removed by the operating system file system code. When trying to rename or delete these files, the iSeries interprets the slashes and quotes as part of the file name and generates an error, CPFA0A9 Object not found, because this naming convention is not valid for the iSeries system.

Note: When this problem is encountered, the best way to remove or access these files is to use the same application that created them.

If the original application is not available or it is not able to remove these objects, there are some other options that can be used. These options include FTP, QShell, iSeries Navigator, IFS tools DLTIFSF, and CleanNames.

FTP:

FTP can be used to rename or delete files and directories with invalid names that contain only standard ANSI characters. For example, FTP can be used to delete or rename files with names containing a backslash (‘\’), but cannot handle names with embedded nulls or Unicode characters.

FTP Commands: REN (rename) and DEL (delete) for files, RMDIR (remove directory) for directories.

Note: Directories must be empty before they can be removed with the RMDIR command.

For example, a file was created in the root of the Integrated File System. The files is called \MYFILE.TXT and must be deleted. Below is a WRKLNK showing how this file looks in the operating system Integrated File System.
Work with Object Links

Directory . . . . : /

Type options, press Enter.
3=Copy 4=Remove 5=Next level 7=Rename 8=Display attributes
11=Change current directory …

Opt Object link Type Attribute Text
QPWXGRB DIR
QPWXGUM DIR
QSR DIR
QSYS.LIB DIR PROD System Library
QTCPTMM DIR
QVGN DIR
RONTEST DIR
\MYFILE.TXT STMF
Snyder DIR
More…
Parameters or command
===>
F3=Exit F4=Prompt F5=Refresh F9=Retrieve F12=Cancel F17=Position to
F22=Display entire field F23=More options

Take the following steps to remove the file:

Note: An FTP session may be started either from the iSeries Command Line (on the same iSeries system or on a different iSeries system) or it may be started from a PC DOS Command Prompt.
1 To start an FTP session to the IBM System i system, on the operating system command line type the following command:

FTP

Press the Enter key. You are prompted to sign on and type your password.
2 Once signed on, change the naming format from operating system to UNIX by issuing the NAMEFMT 1 command (quote site namefmt 1, and press the Enter key). The FTP session should respond with 250 Now using naming format “1”.
3 To change to the root of the Integrated File System, type the following:

CD /

Press the Enter key. Response from the iSeries family system should be 250 “/” is current directory.

If the file is located in a directory or a subdirectory rather than on the root of the Integrated File System, issue CD dirname, and press the Enter key. Response from the iSeries family system should be 250 “/dirname” is current directory.
4 Type the following:

DEL \MYFILE.TXT.

Press the Enter key. The response is 250 Deleted file /\MYFILE.TXT. This also works with the RENAME command.
Note: Remember that this is a UNIX format. Therefore, file names are case-sensitive.

The same steps may be used for removing directories with invalid names (such as \MYDIR). To do so, follow the steps above substituting the RMDIR command in place of the DEL command used in the example. If the directory which has the invalid character in the name contains other DIRs or STMFs, you will probably need to do a REN on the directory with the invalid character in the name and give it a valid name. At this point, you can use normal methods to delete the contents of the directory and remove it as you normally would.

QShell:

QShell can be used to remove some invalid file names, including those that contain a backslash as part of the name. To remove a name with a backslash, escape the character with an additional backslash or double quote the name.

Example: To remove “myfi\le” use ‘rm myfil\\le’ or ‘rm “myfi\le”‘

To use QSHELL to remove the files, do the following:
1 From an operating system command line, type the following:

STRQSH

Press the Enter key.
2 To change directory to the directory containing the invalid file name, type the following: CD mydir
3 To remove the file, type the following: RM “invalid file name”

Note: Double-quotes are required in the RM command.
4 Press F3 to end QSHELL.
iSeries Navigator:

The iSeries Navigator File Systems | Integrated File Systems option can be used to delete or rename files and directories with names that Windows considers invalid (and hence network drives can’t handle). This includes names like *.* or *dir and many special ANSI characters such as the trademark symbol (TM) and so on.

To use iSeries Navigator to remove or rename the files, do the following:
1 Open iSeries Navigator.
2 Expand My Connections.
3 Expand File Systems.
4 Expand Integrated File systems and locate the directory or file containing the invalid file name.
5 Right click on the directory or file name and chose the option to delete or rename the file.

IFSTOOLS DLTIFSF and RNMIFSF:

DLTIFSF and RNMIFSF can be used to delete or rename files or directories with names that contain a backslash.

Example (delete): CALL DLTIFSF ‘[filepath]‘

Example (rename): CALL RNMIFSF PARM(‘[filepathold]‘ ‘[filepathnew]‘

Information about downloading and installing IFSTOOLS is in Rochester Support Center knowledgebase document 19175649, Integrated File System Tools: DEL, DELTREE, CMDALL, CHGAUTALL, CHGOWNALL, QRYIFSLIB, DLTIFSF, RNMIFSF:

CleanNames:

CleanNames is a Java toolbox utility. It is the best option to use when cleaning up thousands of files or when file names include embedded nulls. CleanNames can clean up invalid directory and file names such as: “*.*”, “*name”, “\name”, name with embedded null, name with a Unicode character, and so on. It does not work for names of “.”, “..”, or names that include a forward slash (‘/’).

Command syntax:

CleanNames SystemName TargetDir [option]

SystemName – AS/400 system name as entered in DNS or local hosts table
or the AS/400 system IP address. The name “localhost”
is a valid name when run from the AS/400 jvm.
TargetDir – The directory from which to start work. This is the
directory to delete all files from or the directory
to search for other directories with invalid names.
[option] – The menu option to execute. If not supplied, the user
is prompted for one of following options:

1 Rename all files in the target directory to a valid name.
The names are qfrecov1, qfrecov2, etc…
2 Rename all directories in the target directory to a valid name.
The names are qdrecov1, qdrecov2, etc…”
3 List all objects in the target directory. Prompt to rename.

WRKLNK Option 4=Remove:

The WRKLNK command is not Unicode-enabled. It cannot work with files or directories that have ANSI or Unicode characters that do not exist in the operating system CCSID.

EDTF:

EDTF STMF(/) will list Stream Files and Directories on the root of the IFS. Normal IFS commands (5 to display, and so on) can be used to locate the Stream File with the invalid name.

Use opt 4 to delete file or opt 9 to delete a directory and its contents.

Chris…

Nov 13

FTP Client Version 6.1 available for free 30 day trial

As part of the revitalization of the FTP Security Manager which is now available for download, we took the opportunity to update the FTP client which has the same FTP functionality as the one shipped in the FTP Security Manager without the security controls.

This version provides a much smoother interface to the previous incarnation, it also recognizes the CCSID of the system allowing the conversion of EBCDIC to ASCII to be carried out correctly. We have added some new features which allow the user to switch between remote and local directory listings with the press of a key and removed some of the old screens making navigation of the product much easier. Options have been aligned with the options used when navigating the IFS using the WRKLNK command reducing some of the confusion when first using the product.

This version also brings a new feature which allows the user to edit a local file or display a remote file locally. This is carried out using the IBM supplied EDTF and DSPF commands. We have made many other changes in this release and continue to develop new ones such as SSL support which we hope to have in the next version.

If you want to check it out you can do so with a 30 day free trail which can be downloaded from our members section.

Those who have tried it so far love the ease of use it brings and the ability to remove the practice of moving objects via a PC to get them to the IBM ‘i’ because users prefer a better interface than the IBM FTP Client provides today.

Please take the time to download the product and try it out, we look forward to any comments you have..

Chris…

Nov 13

Audit Journal Entry T OM layout and CCSID 1200

We have been working on the IFS replication process for RAP and came across a problem with the rename process. When a rename of an IFS object takes place a T OM audit entry is generated in the Audit journal when the object is set up for Audit control.

We wanted to be able to to create a RNM command from the information supplied allowing us to run the command against a target IFS with the objective of keeping each side in synch. We coded up the process to carry out the request but it failed every time on the target stating the NEWOBJ parameter was incorrectly formed, we had taken the New Absolute Path information and used it define the new object information. The problem was the Absolute Path has the complete path including the object name so the fact that it started with a ‘/’ caused the request to error. Reading the help text showed us the error of our ways!

This led us to start looking at the New Object Name information, we could use the Old Absolute Path Name for the OBJ part of the command and use the New Object Name for the NEWOBJ part of the command. But when we coded this up we also found the command would fail every time on the target! This time no information was copied from the journal entry into the NEWOBJ part of the command string!

We looked at the data in the journal entry and noticed it was not as we expected (according to the manual it should be a NULL terminated string) but was infact what appeared to be a set of ASCII characters each with a 0x00 character between each one, translation of the ASCII characters did correctly translate to our required information. After some discussions with IBM we found out the data was stored in CCSID 1200, further investigation also showed that this CCSID was wide character based which explained the 0x00 between each of the characters. So we now had the right information to work with!

Initial attempts to code up the conversion from 1200 to 37 failed at various stages, we are using the iconv() API for the conversion which would throw out errors after doing part of the conversion. Again we found out more by trial and error than information from the documentation that when you pass in a wide string to the iconv() API it has to be told the full length of the string as multiples of 2. We had been working out the length of the string using wcslen() and adding 1 for the NULL termination, we needed to add 2 for the NULL termination for the inconv() API to correctly carry out the conversion without errors.

Once we had that figured out, the program correctly formed the command string and it runs with out error on the target system. Having gone through all of that we did however stick with our original solution which was to take the New Absolute Path Name and extract the object name from it. We felt the overhead of running the iconv() process for every request would add to much overhead especially as the data is already provided in the correct CCSID for us to work with.

I have suggested IBM review the detail in the Manual, the information could be misleading to some as it was to us. Our assumption that the CCSID defined at offset 885 only referred to the Old Object Name was incorrect. The Old Object Name is also length specified which the New Object Name is not. So if you are going to work with the IFS audit journal entries please bear this in mind, it could save you a lot of time..

I hope that helps others avoid the silly mistakes we made… Thanks to IBM for the support they provided us in tracking down the real reason for the missing data…

Chris…

Nov 05

Refaced FTP Manager with a few minor changes using Newlook

HI

The following are a couple of pictures of the new refacing done with Newlook. They show the very basic out of the box look with a few changes which we made to replace the Vertical Scroll Bar with an UpDown Control. This is the very first stage of the exercise but shows what is possible after a very short period of time! These can be compared with the original screen shots taken below to show you how a very quickly you can transform existing 5250 screen into fairly reasonable GUI screens. We have a long way to go with the project and we are learning all the time, writing VBScript and Macro’s is not something we have done before so we are seeing a lot of stupid errors creeping in.. But we eventually get the way of doing things and then we can move forward very quickly.

Anyhow here are a couple of the pictures

NewLook Site List

NewLook Site List

NewLook Object Attribute Display

NewLook Object Attribute Display

NewLook Log View

NewLook Log View

NewLook File edit

NewLook File edit

NewLook Directory Display

NewLook Directory Display

If you would like to reface your application lets us know, we would be more than happy to share our experiences so far..

Chris…