I am still trying to get IBM to provide a production level solution for this problem, but in the in the meantime I have been able to take the information supplied from IBM and code around the issues to allow the programs to correctly apply Journal Entries generated on both V6R1 & V5R4 systems.

The solution is not production ready because it requires assumptions to be made that in a real world may not be correct (IBM is still sending me confusing information about the offset placement). The biggest being the state of the ESD pointer, I am assuming if it is NULL then the journal entry was probably created on a V6R1 system and I should use the IBM offsets to get to the data required for the QDBRPLAY API. Otherwise use the pointer I had used before…

This is the test code I originally developed for IBM to show the issue, I am not building all of the data for the QDBRPLAY API just making sure it is available. You will need to add a lot of code to get the journal entries and start the process but this is about the QDBRPLAY API issue where the pointer to additional data is incorrectly positioned in the structure. But if you understand this part of the exercise you can use it to complete your journal entry apply process.

   jrne = space;
   jrne += Jrn_Spc_Hdr->Offset_First_Jrn_Entry;
   Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
   Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
   printf("Num Ents = %d\n",Num_Ents);
   printf("Size of QSYSINC Structure = %d\n",
          sizeof(_Packed struct QdbJrnl_DDL0100));
   for(i = 0; i < Num_Ents; i++) {

      printf("Type = %.2s\n",Jrn_Spc_Ent->Entry_Type);
      printf("Code = %c\n",Jrn_Spc_Ent->Jrn_Code);
      printf("Seq_num found %lld\n",Entry_Seq_Number);
      tmp = jrne;
      /* need to make sure the incomplete data bit is set */
      /* Jrn_Spc_Ent->Incomplete_Data   */
      tmp += Jrn_Spc_Ent->Dsp_Entry_Specific_Data;
      Data = (Qjo_RJNE0100_JE_ESD_t *)tmp;
      ESD_Len = QXXZTOI(Data->Len_ESD,5,0);
      printf("ESD Len = %d\n",ESD_Len);
      memcpy(Input_Template.Base.Journal_Type,
             Jrn_Spc_Ent->Entry_Type,2);
      Input_Template.Base.Journal_Code = Jrn_Spc_Ent->Jrn_Code;
      /* move esddta to the data in case no additional data */
      printf("Referential Constraint %d\n",Jrn_Spc_Ent->Referential_Constraint);
      printf("Trigger %d\n",Jrn_Spc_Ent->Trigger);
      printf("Incomplete data %d\n",Jrn_Spc_Ent->Incomplete_Data);
      printf("Ignore during APYRMV %d\n",Jrn_Spc_Ent->Ignore_during_APYRMV);
      printf("Min ESD %d\n",Jrn_Spc_Ent->Min_ESD);
      printf("File Type Indicator %d\n",Jrn_Spc_Ent->File_Type_Indicator);
      printf("Min Field Boundaries %d\n",Jrn_Spc_Ent->Min_Field_Boundaries);
      printf("Reserved1 %d\n",Jrn_Spc_Ent->Reserved1);
      if(Jrn_Spc_Ent->Incomplete_Data == 1) {
         if(Crt_Usr_Spc(ESD_Dta_SPC,_16MB) != 1) {
            printf("Could not create ESD Space\n");
            return -1;
            }
         QUSPTRUS(ESD_Dta_SPC,
                  &esddta,
                  &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Could not get pointer to ESD Space\n");
            QUSDLTUS(Ent_Req_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               Error_Code.EC.Bytes_Available = 0;
               }
            QUSDLTUS(Jrn_Ent_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               }
            return -1;
            }
         /* set the template for the ESD */
         ESD_Template = (ESD_Tmp_Str_t *)&Data->ESD;
         /* move ptr to UsrSpace for storing data */
         tmp1 = esddta;
         /* move the data to the usrspc from the journal entry */
         memcpy(tmp1,&Data->ESD,(ESD_Len - 16));
         tmp1 += (ESD_Len - 16);
         /* pointer error when hign mem usage */
         printf("ESD Data Length TS %d\n",
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         memcpy(tmp1,ESD_Template->Int_Data,
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");
      /* remove the ESD pointer */
      if(Jrn_Spc_Ent->Ptr_Handle != 0) {
         QjoDeletePointerHandle(&Jrn_Spc_Ent->Ptr_Handle,
                                &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Failed to delete pointer handle\n");
            }
         QUSDLTUS(ESD_Dta_SPC,
                  &Error_Code);
         }
      if(Jrn_Spc_Ent->Dsp_Next_Jrn_Hdr > 0) {
         jrne += Jrn_Spc_Ent->Dsp_Next_Jrn_Hdr;
         Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
         Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
         }
      Start_Seq_Number = Entry_Seq_Number;
      Start_Seq_Number ++;
      }
   } /* else */ 

As explained in the previous post this works fine when applied against a journal receiver created on a V5R4 system. But fell down when trying to map over a receivers entries generated in V6R1. SO I have added the following code in the appropriate place

   if(ESD_Template->Int_Data != NULL) {
      /* if its not NULL I can use it, probably from V5R4?? */ 
            memcpy(tmp1,ESD_Template->Int_Data,
                   ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
            }
         else {
            /* I assume its a V6R1 entry */
            esd_offset = tmp;
            esd_offset += 0x44;
            memcpy(tmp1,esd_offset,ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
            }
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");

Now, while the above code works it is using totally undocumented information so I would say its probably able to change?

If you take the above offset and add it to the start of the ESD information it will actually map to the last 4 bytes of – char DbJrnl_Reserved1[12]; – in the QdbJrnl_DDL0100 structure, I assume that IBM will probably amend the header file to open up that information to help resolve the problem, but it is flawed when addressing a V5R4 entry because its not set in those entries for some reason?

As more information becomes available I will continue to post it here and in the forums for those of you who are affected by this problem.

Chris…

Gentlemen this is a long Post, please make yourself comfortable. I am only posting this because I am totally confused by IBM’s response and fully accept that I am totally stupid and should get a better day job. I have been trying to get to the bottom of this since Tuesday after raising a SEV 1 error with IBM. The course it took is a story on its own, I will share that with those who are interested separately (Dirty Washing)

If you already read the iSeriesNetwork forums, you can skip this one! Its an exact copy of the one I posted in the forums there with just a few spelling errors changed and a couple of presentation enhancements.

The problem is in the journalling area and is based on what I have managed to glean from the manuals supplied by IBM, I have to admit that being English has not helped with the interpretation of the content of the manuals so I may have totally misunderstand what is being said. For those who don’t have English as your first language you may totally miss the interpretation, sorry for that!

OK so first of all lets look at what information I started to work with

Here is the starting place, it is from the Manuals about the use of the QjoREtrieveJournalEntries. This is the part of the structure definitions which describe the RJNE0200_JE_Hdr_t. I have cut down the data for expedience as the rest doesn’t affect the problem I am having.

218(0) DA(0) BIT(1) Referential constraint
218(1) DA(1) BIT(1) Trigger
218(2) DA(2) BIT(1) Incomplete data
218(3) DA(3) BIT(1) Ignored during APYJRNCHG or RMVJRNCHG
218(4) DA(4) BIT(1) Minimized entry specific data
218(5) DA(5) BIT(1) File type indicator
218(6) DA(6) BIT(1) Minimized on field boundaries
218(7) DA(7) BIT(1) Reserved

I had raised a question previously in the forums about how to manipulate the bit fields You can find that post on the iSeriesNetwork forums.

Here is the Output from QSYSINC/H(QJOURNAL) Qjo_RJNE0200_JE_Hdr_t which matches the information above

unsigned int Referential_Constraint:1; /* @ACA @
unsigned int Trigger:1; /* @ACA @
unsigned int Incomplete_Data:1; /* @ACA @
unsigned int Ignore_during_APYRMV:1; /* @ACA @
unsigned int Min_ESD:1; /* @ACA @
unsigned int File_Type_Indicator:1; /* @
unsigned int Min_Field_Boundaries:1; /* @
unsigned int Reserved1:1; /* @ACA @AGC @AKA @

So while the manual states the flags are bits, the C structure states they are Unsigned Int? But that really doesn’t matter either because the C program using the above structure from the QSYSINC/H file returns the correct information.

Here is the quote from the manual about the Incomplete data flag

Incomplete data. Whether this entry has data that must be additionally retrieved using a pointer returned for the missing information. See Use of Pointers within Entry Specific Data for more information. The possible values are:
0 This entry does not have any pointers included.
1 This entry does have pointers included.

The pointer within the text sends the reader to the following information, I left it more or less intact but you could read it online for the full version if needed. I also said this post would be long…

Use of Pointers within Entry Specific Data

There are some journal entries that require additional handling of the journal receiver entry specific data using pointers. This was done to minimize movement of large amounts of data and to facilitate support of tables or database files with large object (LOB) fields. The types of entries that may require pointer support are:

* Any operations on specific records or files (journal code R or F) of tables or database files that include any fields of data type BLOB (binary large object), CLOB (character large object), or DBCLOB (double-byte character large object). See the DB2 UDB for iSeries SQL Programming Concepts and DB2 UDB for iSeries SQL Reference books for more information on these data types.
* Operations related to byte stream file write operations, Journal Code B, Entry type WA. See the Integrated file system information for more information about these journal entries.
* Operations related to data queue send operations, Journal Code Q, Entry types QK and QS. See the Journal management topic for more information on these journal entries.
* Any operations on specific records or files (journal code R or F) of tables or database files resulting in minimized entry specific data when the journal has MINENTDTA specified for the corresponding object type. See the Journal management topic for restrictions and usage of journal entries with minimized entry specific data.

If the incomplete data indicator is returned as a 1, then that indicates that the journal-entry specific data includes a pointer to additional data. Additionally, a pointer handle will be returned with the journal entry. This handle is associated with any allocations required to support the pointer processing.

The pointer must be used by the same process that called this API; it cannot be stored and used by a different process. The pointer can be used for read access only. See the Journal management topic for descriptions of the entry types that may include pointer data. The pointer can be used in the following way:

* It can be used directly to copy the data addressed to some other storage space.
* If the journal entry is a record entry (journal code R), the journal-entry specific data could be used for an update or insert operation to the database file through SQL. See the DB2 UDB for iSeries SQL Reference book for more information.

The pointer handles will be implicitly deleted when the process that requested the journal entries is ended.

These pointers can be used only with the V4R4M0 or later versions of the following languages:

* ILE COBOL
* ILE RPG
* ILE C if the TERASPACE parameter is used when compiling the program. See the WebSphere Development Studio: ILE C/C++ Programmer’s GuideLink to PDF book for more information.

Once the pointer data is used, you must delete the pointer handle to free the handle and any allocations associated with that handle. This can be done by using the Delete Pointer Handle (QjoDeletePointerHandle) API. If the handles are not deleted, the maximum number allowed can be reached, which will prevent further retrieval of journal entries. The deletion must occur from the same process that called the Retrieve Journal Entries (QjoRetrieveJournalEntries) API.

Even if the journal entry data is not used, all pointer handles returned to the user through this interface should be deleted. This is also true when partial journal entry information is returned, even though an error message was returned.

I have included the part of the program which maps the relevant structures over the data, remember these are the same regardless of what OS level the Journal entry I am mapping over comes from.(this point becomes clearer later I hope?)

   jrne = space;
   jrne += Jrn_Spc_Hdr->Offset_First_Jrn_Entry;
   Jrn_Spc_Ent = (Qjo_RJNE0200_JE_Hdr_t *)jrne;
   Entry_Seq_Number = Jrn_Spc_Ent->Seq_Number;
   printf("Num Ents = %d\n",Num_Ents);
   for(i = 0; i < Num_Ents; i++) {
      printf("Type = %.2s\n",Jrn_Spc_Ent->Entry_Type);
      printf("Code = %c\n",Jrn_Spc_Ent->Jrn_Code);
      printf("Seq_num found %lld\n",Entry_Seq_Number);
      tmp = jrne;
      /* need to make sure the incomplete data bit is set */
      /* Jrn_Spc_Ent->Incomplete_Data   */
      tmp += Jrn_Spc_Ent->Dsp_Entry_Specific_Data;
      Data = (Qjo_RJNE0100_JE_ESD_t *)tmp;
      ESD_Len = QXXZTOI(Data->Len_ESD,5,0);
      printf("ESD Len = %d\n",ESD_Len);
      memcpy(Input_Template.Base.Journal_Type,
             Jrn_Spc_Ent->Entry_Type,2);
      Input_Template.Base.Journal_Code = Jrn_Spc_Ent->Jrn_Code;
      /* move esddta to the data in case no additional data */
      printf("Referential Constraint %d\n",Jrn_Spc_Ent->Referential_Constraint);
      printf("Trigger %d\n",Jrn_Spc_Ent->Trigger);
      printf("Incomplete data %d\n",Jrn_Spc_Ent->Incomplete_Data);
      printf("Ignore during APYRMV %d\n",Jrn_Spc_Ent->Ignore_during_APYRMV);
      printf("Min ESD %d\n",Jrn_Spc_Ent->Min_ESD);
      printf("File Type Indicator %d\n",Jrn_Spc_Ent->File_Type_Indicator);
      printf("Min Field Boundaries %d\n",Jrn_Spc_Ent->Min_Field_Boundaries);
      printf("Reserved1 %d\n",Jrn_Spc_Ent->Reserved1);
      if(Jrn_Spc_Ent->Incomplete_Data == 1) {
         if(Crt_Usr_Spc(ESD_Dta_SPC,_16MB) != 1) {
            printf("Could not create ESD Space\n");
            return -1;
            }
         QUSPTRUS(ESD_Dta_SPC,
                  &esddta,
                  &Error_Code);
         if(Error_Code.EC.Bytes_Available > 0) {
            printf("Could not get pointer to ESD Space\n");
            QUSDLTUS(Ent_Req_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               Error_Code.EC.Bytes_Available = 0;
               }
            QUSDLTUS(Jrn_Ent_SPC,
                     &Error_Code);
            if(Error_Code.EC.Bytes_Available > 0) {
               }
            return -1;
            }
         /* set the template for the ESD */
         ESD_Template = (ESD_Tmp_Str_t *)&Data->ESD;
         /* move ptr to UsrSpace for storing data */
         tmp1 = esddta;
         /* move the data to the usrspc from the journal entry */
         memcpy(tmp1,&Data->ESD,(ESD_Len - 16));
         tmp1 += (ESD_Len - 16);
         /* pointer error when hign mem usage */
         printf("ESD Data Length TS %d\n",
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         memcpy(tmp1,ESD_Template->Int_Data,
                ESD_Template->ESD_Hdr.DbJrnl_Apply_Length);
         ESD_Len += ESD_Template->ESD_Hdr.DbJrnl_Apply_Length;
         }
      printf("Would run API here \n");

Couple of points to make, firstly I am using the IBM supplied structures where possible and simply adding the relevant additions such as a Character pointer to the structure which maps the ESD_Data. I could post all of the information but feel it is irrelevant due to the fact that the maps you will see later actually return sensible data in most cases. Secondly if the structures were wrong I would expect other pointer errors to be returned as I move the data around.

So now down to the data I am working with. I had a response from IBM as follows

Hi Chris,
I did not originally work on this issue but tought it would be better if Peter spoke with you about
devellopements response, I will forward it to you:
Developement ran the recreation scenario and can see the D CT entry
on our system. DSPJRN indicates Incomplete data .. No, Therefore there
is no ptr in this jrn entry. Looks like our customer’s code is
expecting there to be one. If the entry is not >32KB you don’t
get a ptr, as documented in infocenter for the D CT entry:
.
Notes:
1. This data does not apply to integrated file system objects.
2. If the data for these entries exceeds 32 KB, then a pointer is
returned to the actual data when the entry is retrieved using an
option to return pointers. If the return pointer option is not
used, then *POINTER is returned for the entry-specific data.
So this not a system code bug, the issue is with the application code.
thanks
Robert

Thank you for using IBM products and support.

So IBM did their debug by taking the journal receiver and journal I had sent and did a DSPJRN against it? I was amazed! I am not using DSPJRN I am using an API? They have the code and they have the object, they insisted on all of that before they would even consider looking at it. Why if they only ran a DSPJRN to come up with their answer (Its the stupid user?) did they need it?

Then

Hi Chris,

Thanks for the emails. I’ve spoken to your local support and they will address your questions on Monday.

Regards, Debbie

IBM Australia

I won’t post my response but to say I am not happy after having worked non stop on trying to find out where I am making my stupid assumptions and why IBM must be right is not going to get past the rules of decency… Why they would not talk to me direct is also comical…

So having been told I am an idiot and I have wait until Monday to be told why, I decided to review all of the information I have found and shared with IBM. This post is the last part of this until Monday unless someone can help me see the error of my ways.

First of all you will see IBM is dismissing my problem as a User problem because DSPJRN shows the Incomplete Data Flag is set to No? This made me think Oh my God what an Idiot! Why did I make such a stupid mistake! So I went back through my code to see why..

Based on my reading of the manual data I assumed that if the Incomplete Data Flag is set there should be a Pointer in the returned Journal Entry which is pointing to the data to be retrieved, that pointer should therefore not be NULL? I mean whats the point in saying yes there is a pointer in this data but it may be NULL?? Did I make the biggest fundamental mistake there?

So I went through the scenario’s AGAIN. Remember (its been a long story so far..) that this only seems to occur when the journal entry I am working with is generated on a V6R1 system, a journal entry for the exact same process on a V5R4 system actually works OK. All I am doing as can be seen from the code above is trying to create a file from the data contained in a journal entry The process created a D CT entry which I am using the QDBRPLAY API to replay…

This information is from the system running V5R4. The data in the journal entry is therefore created on a V5R4 system (clear?)

First I looked for an entry which would describe my problem and displayed the information in the same manner as Rochester appears to have. I found an entry in TST0000002 which is attached to #tstjrn/tstjrn.

Here is the output

Display Journal Entries

Journal . . . . . . : TSTJRN Library . . . . . . : #TSTJRN
Largest sequence number on this screen . . . . . . : 00000000000000000013
Type options, press Enter.
5=Display entire entry

Opt Sequence Code Type Object Library Job Time
3 J PR CHRISH#VIB 14:25:33
4 J JR TST0000002 #TSTRCV CHRISH#VIB 14:25:33
5 I UE #DTA QSYS CHRISH#VIB 14:26:39
6 J XP F #DTA CHRISH#VIB 14:26:56
7 I UE #DTA QSYS CHRISH#VIB 14:26:56
8 D CT F #DTA CHRISH#VIB 14:26:56
9 D JF F #DTA CHRISH#VIB 14:26:56
10 F JM F #DTA CHRISH#VIB 14:26:56
11 D MA F #DTA CHRISH#VIB 14:26:56
12 F MC F #DTA CHRISH#VIB 14:26:56
13 J NR CHRISH#VIB 14:27:23

Bottom
F3=Exit F12=Cancel

Display the specific journal entry for the CT

Display Journal Entry

Object . . . . . . . : F Library . . . . . . : #DTA
Member . . . . . . . :
Incomplete data . . : No Minimized entry data : *NONE
Sequence . . . . . . : 8
Code . . . . . . . . : D – Database file operation
Type . . . . . . . . : CT – Create database file

Entry specific data
Column *…+….1….+….2….+….3….+….4….+….5
00001 ‘F #DTA ‘
00051 ‘ µ Ø  Z 1091117′
00101 ’142656 9 PF 00°Ø ¨³’
00151 ‘¹ {Y N *CHANGE ‘
00201 ‘ 1091117142656 ‘
00251 ‘ ‘
00301 ‘ DN ¯ Y ‘
More…
Press Enter to continue.

F3=Exit F6=Display only entry specific data
F10=Display only entry details F12=Cancel F24=More keys