Discussion:
Primary & secondary space values in DFSORT's dynamic work space allocation.
(too old to reply)
Peter Hunkeler
2018-07-04 19:36:24 UTC
Permalink
Some background, first.

I was asked to help a COBOL application calling DFSORT internally via INPUT / OUTPUT PRODECURE (E15/E35) interface. The input data size is unknown, but varying greatly. FILESZ cannot be supplied. So, DFSORT has no way to calculate the required disk work space.


The application was asked to change from JCL allocated SORTWKnn to dynamically allocated sort work space.


DFSORT options DYNALLOC, DYNAPCT, and DYNSPC are the options used to help DFSORT optimally allocate the sort work space. Some extracts from the manual are listed below.


I read the relevant parts in the Installation & Customization, the Application Programmer, and the Tuning Guide, but still I'm not sure I understand how and when the work space data set are allocated. The disk space allocated is of special interest here.


Suppose: DYNALLOC=(,8),DYNAPCT=50,DYNSPC=2400


Now here are the statements with a lot of uncertainty, and question marks. I'd appreciate confirmation, or correction where I'm wrong.


a) DFSORT will allocate 8 initial work data sets with primary space, and 50% of that, i.e. 4 reserve data sets with zero space initially. The total amount of primary space is the equivalent of 2400 MB. The manual says this is the total over *all* work data sets.



b) So, the primary space for each of the 12 work data sets is 200MB, but only the 8 initial ones are allocated with that primary amount. The 4 reserved data sets are initially allocated with 0 space, but 200 MB will be allocated once DFSORT decides it needs them.




c) The secondary space is said to be 20% of the primary, i.e. 480MB. It is not clear to me, whether this means any secondary extent is 480MB, or whether the secondary value is 480MB / 12, i.e. 40MB. This greatly influences the theoretical total amount of workspace.




d) When DFSORT decides it needs more space, will it extend one, some, or all of the additional work data sets? At this time, the initial work data set have probably been extended to their maximum size (1 x primary + 15 x secondary). Will the additional data sets be extended by the primary amount only, and will grow later as needed?




e) The first extension amount used with d) will be the primary amount calculated above, i.e. 200MB, right? Later, these additional data sets can be expanded by the secondary extent value, up to 15 times. Correct?





Extracts from the Application Programmer's Guide:

DYNAPCT=x

specifies additional work data sets to be dynamically allocated with zero space. DFSORT only extends these data sets when necessary to complete a sort application.

x specifies the number of additional work data sets (y) as a percentage of the maximum number of dynamically allocated work data sets (DYNALLOC/DYNALOC n value) in effect.

DYNSPC=n

DYNSPC=n temporarily overrides the DYNSPC installation option, which specifies the total default primary space allocation for all of the dynamically allocated work data sets when the input file size is unknown.

..., DFSORT uses the DYNSPC value in effect as the approximate amount of primary space. DFSORT uses 20% of the primary space as secondary space. Although the primary space is always allocated, secondary space (up to 15 extents) is only allocated as needed.

n specifies the total default primary space, in megabytes, to be allocated for all dynamically allocated work data sets (n is not the primary space for each data set).
--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Hobart Spitz
2018-07-09 20:52:42 UTC
Permalink
It might be worth pointing out what might be a little known fact. Any
allocation, primary or secondary, can be done in up to 5 extents. That
means that, on a highly fragmented pack, SPACE=(CYL,(5,5)) might only get
16 cylinders (in 16 extents) before running out of extends. Fragmentation
at this level is unusual on modern systems, but I wouldn't assume that all
allocations were in single extents. If you have significant fragmentation,
perhaps because work packs are not managed like other packs, you may be
dealing with a different problem then you think.

If you think specifying FILSZ might help, have you considered adding a REXX
EXEC that issues a CALL LISTDSI or a SQL SELECT COUNT(*) ... and that
calculates a crude estimate of space from that information. The result
could then be passed to the COBOL program, either as a parm or in a VIO
dataset, and the COBOL program could pass that to sort.

Have you tried it and are getting some kind of negative result, or are you
just in the planning stage?



OREXXMan
JCL is the buggy whip of 21st century computing. Stabilize it.
Put Pipelines in the z/OS base. Would you rather process data one
character at a time (Unix/C style), or one record at a time?
IBM has been looking for an HLL for program products; REXX is that language.
Post by Peter Hunkeler
Some background, first.
I was asked to help a COBOL application calling DFSORT internally via
INPUT / OUTPUT PRODECURE (E15/E35) interface. The input data size is
unknown, but varying greatly. FILESZ cannot be supplied. So, DFSORT has no
way to calculate the required disk work space.
The application was asked to change from JCL allocated SORTWKnn to
dynamically allocated sort work space.
DFSORT options DYNALLOC, DYNAPCT, and DYNSPC are the options used to help
DFSORT optimally allocate the sort work space. Some extracts from the
manual are listed below.
I read the relevant parts in the Installation & Customization, the
Application Programmer, and the Tuning Guide, but still I'm not sure I
understand how and when the work space data set are allocated. The disk
space allocated is of special interest here.
Suppose: DYNALLOC=(,8),DYNAPCT=50,DYNSPC=2400
Now here are the statements with a lot of uncertainty, and question marks.
I'd appreciate confirmation, or correction where I'm wrong.
a) DFSORT will allocate 8 initial work data sets with primary space, and
50% of that, i.e. 4 reserve data sets with zero space initially. The total
amount of primary space is the equivalent of 2400 MB. The manual says this
is the total over *all* work data sets.
b) So, the primary space for each of the 12 work data sets is 200MB, but
only the 8 initial ones are allocated with that primary amount. The 4
reserved data sets are initially allocated with 0 space, but 200 MB will be
allocated once DFSORT decides it needs them.
c) The secondary space is said to be 20% of the primary, i.e. 480MB. It is
not clear to me, whether this means any secondary extent is 480MB, or
whether the secondary value is 480MB / 12, i.e. 40MB. This greatly
influences the theoretical total amount of workspace.
d) When DFSORT decides it needs more space, will it extend one, some, or
all of the additional work data sets? At this time, the initial work data
set have probably been extended to their maximum size (1 x primary + 15 x
secondary). Will the additional data sets be extended by the primary amount
only, and will grow later as needed?
e) The first extension amount used with d) will be the primary amount
calculated above, i.e. 200MB, right? Later, these additional data sets can
be expanded by the secondary extent value, up to 15 times. Correct?
DYNAPCT=x
specifies additional work data sets to be dynamically allocated with zero
space. DFSORT only extends these data sets when necessary to complete a
sort application.
x specifies the number of additional work data sets (y) as a percentage of
the maximum number of dynamically allocated work data sets
(DYNALLOC/DYNALOC n value) in effect.
DYNSPC=n
DYNSPC=n temporarily overrides the DYNSPC installation option, which
specifies the total default primary space allocation for all of the
dynamically allocated work data sets when the input file size is unknown.
..., DFSORT uses the DYNSPC value in effect as the approximate amount of
primary space. DFSORT uses 20% of the primary space as secondary space.
Although the primary space is always allocated, secondary space (up to 15
extents) is only allocated as needed.
n specifies the total default primary space, in megabytes, to be allocated
for all dynamically allocated work data sets (n is not the primary space
for each data set).
--
Peter Hunkeler
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-07-10 13:54:55 UTC
Permalink
It might be worth pointing out what might be a little known fact....[snip]
Thanks, I do understand this. Nevertheless, good point. Worth to be found in the archives.
If you think specifying FILSZ might help,... [snip]
I do think so, indeed, but I talked to the application people and the have now way to tell and pass to sort in this case.


BTW, I worth a test program meanwhile and found that the manual and the reality diverge. I'm pursuing this offline and will report back once I now more.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-07-11 12:45:47 UTC
Permalink
BTW, I wrote a test program meanwhile and found that the manual and the reality diverge. I'm pursuing this offline and will report back once I know more.
I got a response from Kolusu. We're gonna open a PMR for this.
--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Vernooij, Kees - KLM , ITOPT1
2018-07-11 13:08:15 UTC
Permalink
What is in fact the problem with this job, that the SORTWK specification should be changed from JCL to dynamic?

It seems, that you try to provide the much varying workspace, where in fact, the job should be able to get its space anyhow.

What is the problem with specifying the historical 90% space consumption and allow for 50 or 100% growth, in JCL?
When needed, you must have the largest amount of space available, so why not make it available to every run of the job? It is only temporary, seconds/minutes/qtrs/hours, not permanent.

Kees.
-----Original Message-----
Behalf Of Peter Hunkeler
Sent: 04 July, 2018 21:36
Subject: Primary & secondary space values in DFSORT's dynamic work space
allocation.
Some background, first.
I was asked to help a COBOL application calling DFSORT internally via
INPUT / OUTPUT PRODECURE (E15/E35) interface. The input data size is
unknown, but varying greatly. FILESZ cannot be supplied. So, DFSORT has
no way to calculate the required disk work space.
The application was asked to change from JCL allocated SORTWKnn to
dynamically allocated sort work space.
DFSORT options DYNALLOC, DYNAPCT, and DYNSPC are the options used to
help DFSORT optimally allocate the sort work space. Some extracts from
the manual are listed below.
I read the relevant parts in the Installation & Customization, the
Application Programmer, and the Tuning Guide, but still I'm not sure I
understand how and when the work space data set are allocated. The disk
space allocated is of special interest here.
Suppose: DYNALLOC=(,8),DYNAPCT=50,DYNSPC=2400
Now here are the statements with a lot of uncertainty, and question
marks. I'd appreciate confirmation, or correction where I'm wrong.
a) DFSORT will allocate 8 initial work data sets with primary space, and
50% of that, i.e. 4 reserve data sets with zero space initially. The
total amount of primary space is the equivalent of 2400 MB. The manual
says this is the total over *all* work data sets.
b) So, the primary space for each of the 12 work data sets is 200MB, but
only the 8 initial ones are allocated with that primary amount. The 4
reserved data sets are initially allocated with 0 space, but 200 MB will
be allocated once DFSORT decides it needs them.
c) The secondary space is said to be 20% of the primary, i.e. 480MB. It
is not clear to me, whether this means any secondary extent is 480MB, or
whether the secondary value is 480MB / 12, i.e. 40MB. This greatly
influences the theoretical total amount of workspace.
d) When DFSORT decides it needs more space, will it extend one, some, or
all of the additional work data sets? At this time, the initial work
data set have probably been extended to their maximum size (1 x primary
+ 15 x secondary). Will the additional data sets be extended by the
primary amount only, and will grow later as needed?
e) The first extension amount used with d) will be the primary amount
calculated above, i.e. 200MB, right? Later, these additional data sets
can be expanded by the secondary extent value, up to 15 times. Correct?
DYNAPCT=x
specifies additional work data sets to be dynamically allocated with
zero space. DFSORT only extends these data sets when necessary to
complete a sort application.
x specifies the number of additional work data sets (y) as a percentage
of the maximum number of dynamically allocated work data sets
(DYNALLOC/DYNALOC n value) in effect.
DYNSPC=n
DYNSPC=n temporarily overrides the DYNSPC installation option, which
specifies the total default primary space allocation for all of the
dynamically allocated work data sets when the input file size is
unknown.
..., DFSORT uses the DYNSPC value in effect as the approximate amount of
primary space. DFSORT uses 20% of the primary space as secondary space.
Although the primary space is always allocated, secondary space (up to
15 extents) is only allocated as needed.
n specifies the total default primary space, in megabytes, to be
allocated for all dynamically allocated work data sets (n is not the
primary space for each data set).
--
Peter Hunkeler
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
********************************************************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
********************************************************

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-07-11 14:17:22 UTC
Permalink
Post by Vernooij, Kees - KLM , ITOPT1
What is in fact the problem with this job, that the SORTWK specification should be changed from JCL to dynamic?
[snip]


I totally agree with all you write.


I don't know, and honestly I don't understand why they were asked to do change. I was only contacted by the desperate developer when he recognized the change could lead to production job abends. This was just before the software delivery cycle.


I tried to understand how the parameters for the dynamic allocation work, so as to be able to give good recommendation to them. By testing, I found that the DYNAPCT creates some reserve work space data sets, but the total space of *all* the reserve data sets is roughly the space of a *single* initial work data set. Doesn't make sense to me, and IBM seems to agree. Therefore the PMR.




--
Peter Hunkeler





----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Vernooij, Kees - KLM , ITOPT1
2018-07-11 14:25:07 UTC
Permalink
Ok, I'm happy to see we have the same view on the (ahum) 'problem'.

Kees.
-----Original Message-----
Behalf Of Peter Hunkeler
Sent: 11 July, 2018 16:17
Subject: AW: Re: Primary & secondary space values in DFSORT's dynamic
work space allocation.
Post by Vernooij, Kees - KLM , ITOPT1
What is in fact the problem with this job, that the SORTWK
specification should be changed from JCL to dynamic?
[snip]
I totally agree with all you write.
I don't know, and honestly I don't understand why they were asked to do
change. I was only contacted by the desperate developer when he
recognized the change could lead to production job abends. This was just
before the software delivery cycle.
I tried to understand how the parameters for the dynamic allocation
work, so as to be able to give good recommendation to them. By testing,
I found that the DYNAPCT creates some reserve work space data sets, but
the total space of *all* the reserve data sets is roughly the space of a
*single* initial work data set. Doesn't make sense to me, and IBM seems
to agree. Therefore the PMR.
--
Peter Hunkeler
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
********************************************************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
********************************************************

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Elardus Engelbrecht
2018-07-11 13:30:16 UTC
Permalink
Post by Vernooij, Kees - KLM , ITOPT1
What is in fact the problem with this job, that the SORTWK specification should be changed from JCL to dynamic?
It seems, that you try to provide the much varying workspace, where in fact, the job should be able to get its space anyhow.
What is the problem with specifying the historical 90% space consumption and allow for 50 or 100% growth, in JCL?
When needed, you must have the largest amount of space available, so why not make it available to every run of the job? It is only temporary, seconds/minutes/qtrs/hours, not permanent.
Ok, time for me to jump in (despite just now posted the note about a PMR):

Your questions are good questions!

Perhaps the OP should also put in the "estimate" on how big the input for all the sorting work is in other terms.

Something like this: Have REGION=<really large> and OPTION DYNSPC=512,SIZE=E999999999,MAINSIZE=MAX (which is working for my hungry sort jobs.)

This will help DFSORT to try to allocate enough pool of work datasets to play with.

This is what I mostly do, I specify really large SORTWKnn and/or really large dynamic sort space. YMMV.

Of course, you should check your SMS setup so you have lots of free space.

If you still can't resolve your sort, try splitting up your sort input and/or just do a subselection.

Groete / Greetings
Elardus Engelbrecht

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-07-11 14:22:49 UTC
Permalink
Post by Elardus Engelbrecht
Perhaps the OP should also put in the "estimate" on how big the input for all the sorting work is in other terms.
Something like this: Have REGION=<really large> and OPTION DYNSPC=512,SIZE=E999999999,MAINSIZE=MAX (which is working for my hungry sort jobs.)
The OP (me :-) wrote in the initial post that the application *cannot* provide an estimate.


I did not write about the fact that DFSORT sometimes decides it cannot do in-memory sorting because it would hurt the overall system health. I did not because that is not part of my questions. Be assured I do understand the in-memory options DFSORT provides.




--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Elardus Engelbrecht
2018-07-11 14:45:21 UTC
Permalink
Post by Peter Hunkeler
Post by Elardus Engelbrecht
Perhaps the OP should also put in the "estimate" on how big the input for all the sorting work is in other terms.
Something like this: Have REGION=<really large> and OPTION DYNSPC=512,SIZE=E999999999,MAINSIZE=MAX (which is working for my hungry sort jobs.)
The OP (me :-) wrote in the initial post that the application *cannot* provide an estimate.
Sorry, I forgot about that 'cannot provide estimate'. Thanks for sorting me out. ;-)
Post by Peter Hunkeler
I did not write about the fact that DFSORT sometimes decides it cannot do in-memory sorting because it would hurt the overall system health. I did not because that is not part of my questions. Be assured I do understand the in-memory options DFSORT provides.
I see your dilemma. I hope you have great success with that PMR. Let us all know how that PMR is faring.

Good luck! All of the very best to you.

Groete / Greetings
Elardus Engelbrecht

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Loading...