In the documentation regarding a FIXED division with RULES(ANS) when one operand is

a scaled FIXED DECIMAL value, we find this for the rules of calculating the resulting precision

when the dividend is FIXED BIN and the divisor is FIXED DECIMAL:

It would be (15,13) if the LRM were correct. But sadly, it isn't.

For the division of 2 FIXED DEC, the book is correct in that the result scale factor is N - p1 + q1 - q2

For the division of FIXED BIN by FIXED DEC, the corresponding result should be the same but with p1 and q1 changed from BIN to DEC. q1 is zero so that is unchanged, and p1 would be CEIL(p1/3.32) which in this case would be 5. The result scale would then be 15 -5 + 0 - 2 or 8

Hi Peter!

 Thanks for that!   But - I'm not sure that is right yet??

 If you convert the FIXED BIN to a FIXED DEC to perform the division, then shouldn't the 

rule be1 + CEIL(p1/3.32)  -  not CEIL(p1/3.32) (from page 83 of the doc?)   

 If that is the case, shouldn't the result be 15-6+0-2 or - 7 ?

 If the converted precision isn't 1 + CEIL(p1/3.32) - then... uh... is there some motivation for it to be different in this situation?

  - Dave

The convert precision for p1 would be what the LRM calls w as in: w = CEIL(p1/3.32) and that is 5

Thanks Peter!

 Is there some reason it's not "r"  as in  r = 1 + CEIL(p1*3.32) ?   Just wondering what the motivating

factor might be in that choice?   it kinda seems to me if we follow the logic of applying the conversion

to common derived type; then the FIXED BIN would go thru the normal conversion to FIXED DEC and

we would arrive at "r" instead of "w".

  And, my next question is going to be about the choice of "N-q1" for the precision of the result

if the operands were reversed...  is there some mathematical reason a different-than-standard rule

is applied in this case?

  Just wondering what the motivating factors may have been in making these decisions...

    - Thanks!

The  r = 1 + CEIL(p1*3.32) is for converting from DEC to BIN. The CEIL(p/3.32) is for converting from BIN to DEC

And in this case, we are converting the FIXED BIN numerator to FIXED DEC

Hi Peter,

 Hmm - this is what the doc says (page 83 if the V5R2 language ref):

   Target: FIXED DECIMAL(p2,q2)

    Source:

       FIXED BINARY (p1,q1)

       The precision of the result is p2 = 1 + CEIL(p1/3.32) and q2=CEIL(ABS(q1 / 3.32))*SIGN(q1).

So... wouldn't a conversion from FIXED BIN to FIXED DEC mean that the resulting precision

is 1 + CEIL(p1/3.32) ?  If not, is there a reason it wouldn't be?  Maybe some fancy arithmetic/computational reason?

I mean - my naive thought would be that the FIXED BIN would be converted to the (common derived type) of FIXED

DEC following the conversion rules, and then this simply becomes a FIXED DEC / FIXED DEC operation with the 

resulting precision calculated following those rules...

I'm wondering if there is some advantage for not following that scheme - maybe preservation of bits in the computation

or something???

The text you cite on page 83 applies to conversions implied in DEC(x), FIXEDDEC(x), CHAR(x), etc (where x is FIXED BIN). But it does not apply in arithmetic operations: there, if FIXED BIN is converted to FIXED DEC, the intermediate precision is CEIL(p/3.32) and this is either v or w in the terminology of table 26.

The multiplication column is correct, but the division column is incorrect not only in row 3 as you pointed out at the beginning of this thread, but also in row 2: the stated value N-q1 makes little sense (especially given the box above it where the result is N-p1+q1-q2. The box in row 2 should be N-p1+q1

Thanks Peter!
 

  That's the issue I don't quite understand - why would the intermediate precision be any different than the normal

 conversion dictates?

  I think, for example, if you consider the precision results when RULES(IBM) is applied, the results are directly

 derived from the substitution of the converted precision - which would be following the standard (the operands

 are converted to the common derived type and then the operation is performed.)     I went thru the steps, and

 neglecting the problem of producing an intermediate value larger than M the formulas seem to directly precede from

 that rule.

  So, why would that not be the case for RULES(ANS) ?  

  For example, consider the situation of FIXED DEC(p1,q1) / FIXED BIN(p2,q2) under RULES(IBM), if you

 apply the conversion of the FIXED DEC(p1,q1) to a FIXED BIN, you'll get a FIXED BIN with the 

 precision of ( min(M, 1+CEIL(p1*3.32)), CEIL(ABS(q1*3.32)*SIGN(q1) ) ).  If you then substitute that

 into the formula for FIXED BIN / FIXED BIN (neglecting the min()) you have p = M,

 q = M - (1+CEIL(p1*3.32)) + CEIL(ABS(q1*3.32)*SIGN(q2)) - q2; which matches the documented

 precision results for FIXED DEC / FIXED BIN (under RULES(IBM)) 

 The same works for FIXED BIN(p1,q1) / FIXED DEC(p2,q2).

  I was just assuming since that approach is taken for RULES(IBM) - a similar approach would

 be had for RULES(ANS) ?     It seems like it's not... so, is there some reason?  If that approach

 isn't taken, wouldn't there be a risk of loosing precision?

     - Thanks

   - Dave R. -

There is no loss of precision since RULES(ANS) rules out non-zero scale factors for FIXED BIN. I see the logic in what you're saying, but the v and w values do suffice, and at this point, this is what it is.