Maximum size of a PDSE library?

[Farley, Peter x23353]

I read in DFSMS "Using Datasets" that a PDSE can have up to 123 extents, but I am not finding any specific limits on total PDSE dataset size.

Specific questions for non-program-object ("data") PDSE's:

1. Are PDSE's limited to 4Gb total size?
2. Can PDSE's be allocated as Address Extended files on an EAV volume?
3. If allocated as Extended, can PDSE's exceed the 65535 total cylinders limit?
4. Is there specific documentation that lists the actual maximum space limits for PDSE's?

We run z/OS V2.1 if that makes any difference.

TIA for any answers you can provide.

Peter
--

This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Tom Marchant]

Post by Farley, Peter x23353
2. Can PDSE's be allocated as Address Extended files on an EAV volume?

Yes.
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_1.13.0/com.ibm.zos.r13.idak100/eav3pl.htm

[Farley, Peter x23353]

Thanks for the reference, Tom. But what about total PDSE size? Can it exceed 4Gb uncompressed? Obviously compression is a viable option too, but it would be helpful to know the actual cylinder/Gb size limit (assuming there is one).

Peter

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Tom Marchant
Sent: Friday, February 17, 2017 10:25 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Maximum size of a PDSE library?

Post by Farley, Peter x23353
2. Can PDSE's be allocated as Address Extended files on an EAV volume?

Yes.
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_1.13.0/com.ibm.zos.r13.idak100/eav3pl.htm

--


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Mike Schwab]

I think a PDSE can only exist on one volume, so a 1TB limit (so far).
64K track limit does not apply.
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieab600/xddspace.htm

On Fri, Feb 17, 2017 at 10:08 AM, Farley, Peter x23353

Post by Farley, Peter x23353
Thanks for the reference, Tom. But what about total PDSE size? Can it exceed 4Gb uncompressed? Obviously compression is a viable option too, but it would be helpful to know the actual cylinder/Gb size limit (assuming there is one).
Peter
-----Original Message-----
Sent: Friday, February 17, 2017 10:25 AM
Subject: Re: Maximum size of a PDSE library?

2. Can PDSE's be allocated as Address Extended files on an EAV volume?

Yes.
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_1.13.0/com.ibm.zos.r13.idak100/eav3pl.htm
--
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

[R.S.]

Post by Farley, Peter x23353
Thanks for the reference, Tom. But what about total PDSE size? Can it exceed 4Gb uncompressed? Obviously compression is a viable option too, but it would be helpful to know the actual cylinder/Gb size limit (assuming there is one).

It's complex a little bit
PDSE was *never* constrained to 4GB or 64k TRK.
PDSE was and still is constrained to one volume. It can be 3390-9, so
called mod-27 and mod-54 and EAV as well. The limit for EAV is 1TB now,
it can be limited by your DASD configuration.

There is also a limit for PDSE member size. For PDSE v.1 it is 15 728
639 records (approx. 1.2GB for FB 80). However since z/OS 2.1 there is
v.2 of PDSE with limit for member at 2 146 435 071 records (approx. 171
GB for FB 80).

There is also a limit for number of members, it's 522 236.

For record purposes, limits of regular PDS are: single volume, 64k TRK
per (single) volume). Member is limited only by the size of PDS, number
of members are limited by directory size. Huge directory provides really
poor performance.
--
Radoslaw Skorupka
Lodz, Poland






---
Treść tej wiadomości może zawierać informacje prawnie chronione Banku przeznaczone wyłącznie do użytku służbowego adresata. Odbiorcą może być jedynie jej adresat z wyłączeniem dostępu osób trzecich. Jeżeli nie jesteś adresatem niniejszej wiadomości lub pracownikiem upoważnionym do jej przekazania adresatowi, informujemy, że jej rozpowszechnianie, kopiowanie, rozprowadzanie lub inne działanie o podobnym charakterze jest prawnie zabronione i może być karalne. Jeżeli otrzymałeś tę wiadomość omyłkowo, prosimy niezwłocznie zawiadomić nadawcę wysyłając odpowiedź oraz trwale usunąć tę wiadomość włączając w to wszelkie jej kopie wydrukowane lub zapisane na dysku.

This e-mail may contain legally privileged information of the Bank and is intended solely for business use of the addressee. This e-mail may only be received by the addressee and may not be disclosed to any third parties. If you are not the intended addressee of this e-mail or the employee authorized to forward it to the addressee, be advised that any dissemination, copying, distribution or any other similar activity is legally prohibited and may be punishable. If you received this e-mail by mistake please advise the sender immediately by using the reply facility in your e-mail software and delete permanently this e-mail including any copies of it either printed or saved to hard drive.

mBank S.A. z siedzibą w Warszawie, ul. Senatorska 18, 00-950 Warszawa, www.mBank.pl, e-mail: ***@mBank.pl
Sąd Rejonowy dla m. st. Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego, nr rejestru przedsiębiorców KRS 0000025237, NIP: 526-021-50-88. Według stanu na dzień 01.01.2016 r. kapitał zakładowy mBanku S.A. (w całości wpłacony) wynosi 168.955.696 złotych.


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Vernooij, Kees - KLM , ITOPT1]

Post by R.S.
Huge directory provides really
poor performance.

Unless cached by LLA or another directory caching product.

Post by R.S.
--
Radoslaw Skorupka
Lodz, Poland

********************************************************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
********************************************************


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

There is also a limit for PDSE member size. For PDSE v.1 it is 15 728 639 records (approx. 1.2GB for FB 80). However since z/OS 2.1 there is v.2 of PDSE with limit for member at 2 146 435 071 records (approx. 171 GB for FB 80).

I believe the tokes used by NOTE and POINT represents logical
records, not blocks. So if one is processing multiple members,
the programmer should first POINT to the member, then POINT to
the record? Can one still add one to the TTRZ to POINT to the
next block (whatever a "block" means)?

For record purposes, limits of regular PDS are: single volume, 64k TRK per (single) volume). Member is limited only by the size of PDS, number of members are limited by directory size. Huge directory provides really poor performance.

Ouch! I would hope for no worse than n log n.
https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_algorithm

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Farley, Peter x23353]

Many thanks for the follow-up Radoslaw. I am pursuing the issue of creating an EAV of sufficient size for our purposes with our Storage Admins now. Unfortunately I do not know the limits (or even the HW vendor) of our DASD system, so I do not yet know what we will be able to do.

Fortunately our immediate issue is not member size but total dataset size, so using a V2 PDSE won’t be needed until we determine whether to use member generations or not. That's a different can of worms.

Peter

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of R.S.
Sent: Monday, February 20, 2017 8:47 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Maximum size of a PDSE library?

Post by Farley, Peter x23353
Thanks for the reference, Tom. But what about total PDSE size? Can it exceed 4Gb uncompressed? Obviously compression is a viable option too, but it would be helpful to know the actual cylinder/Gb size limit (assuming there is one).

It's complex a little bit
PDSE was *never* constrained to 4GB or 64k TRK.
PDSE was and still is constrained to one volume. It can be 3390-9, so
called mod-27 and mod-54 and EAV as well. The limit for EAV is 1TB now,
it can be limited by your DASD configuration.

There is also a limit for PDSE member size. For PDSE v.1 it is 15 728
639 records (approx. 1.2GB for FB 80). However since z/OS 2.1 there is
v.2 of PDSE with limit for member at 2 146 435 071 records (approx. 171
GB for FB 80).

There is also a limit for number of members, it's 522 236.

For record purposes, limits of regular PDS are: single volume, 64k TRK
per (single) volume). Member is limited only by the size of PDS, number
of members are limited by directory size. Huge directory provides really
poor performance.

[Tom Marchant]

Post by Vernooij, Kees - KLM , ITOPT1

Post by R.S.
Huge directory provides really
poor performance.

Unless cached by LLA or another directory caching product.

I'm not sure that helps much.

The poor performance that Radoslaw mentioned is more on directory updates.
If you create a member (insert a directory entry), the last entry will probably
be pushed off the end of the directory entry block. That entry will be moved
to the next block, and probably push the last entry off that block. This
process will continue to the end of the directory.

Similarly, if a member is deleted, the space that remains at the end of the block
will probably be big enough to hold the first entry on the next track. If it is, that
entry will be moved, leaving room at the end of that block. Again, if the first
entry on the block after that one will fit, and it probably will, it is also moved.
Again, this process continues to the end of the directory.

The directory is always maintained in member name order, and all of the directory
blocks except the last one are always kept as full as possible.

For directory searches, SEARCH KEY HIGH OR EQUAL is used, allowing the
correct block to be read with a minimum of CPU processing and data transfer.

[Jesse 1 Robinson]

Although HFS is now deprecated in favor of ZFS, there are still a lot of HFS around, so a note might be in order. HFS is a special implementation of PDSE that is *not* subject to the single-volume restriction. We hit a problem early on with multivolume HFS when allocation failed despite doc to the contrary. IBM discovered that allocation logic was checking data set attributes in the wrong order. They were checking for 'PDSE' attribute before 'HFS' attribute and rejecting multivolume. The APAR fix reversed the order of checks, so multivolume HFS was OK.

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
***@sce.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Farley, Peter x23353
Sent: Monday, February 20, 2017 8:49 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: (External):Re: Maximum size of a PDSE library?

Many thanks for the follow-up Radoslaw. I am pursuing the issue of creating an EAV of sufficient size for our purposes with our Storage Admins now. Unfortunately I do not know the limits (or even the HW vendor) of our DASD system, so I do not yet know what we will be able to do.

Fortunately our immediate issue is not member size but total dataset size, so using a V2 PDSE won’t be needed until we determine whether to use member generations or not. That's a different can of worms.

Peter

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of R.S.
Sent: Monday, February 20, 2017 8:47 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Maximum size of a PDSE library?

Post by Farley, Peter x23353
Thanks for the reference, Tom. But what about total PDSE size? Can it exceed 4Gb uncompressed? Obviously compression is a viable option too, but it would be helpful to know the actual cylinder/Gb size limit (assuming there is one).

It's complex a little bit
PDSE was *never* constrained to 4GB or 64k TRK.
PDSE was and still is constrained to one volume. It can be 3390-9, so called mod-27 and mod-54 and EAV as well. The limit for EAV is 1TB now, it can be limited by your DASD configuration.

There is also a limit for PDSE member size. For PDSE v.1 it is 15 728
639 records (approx. 1.2GB for FB 80). However since z/OS 2.1 there is
v.2 of PDSE with limit for member at 2 146 435 071 records (approx. 171 GB for FB 80).

There is also a limit for number of members, it's 522 236.

For record purposes, limits of regular PDS are: single volume, 64k TRK per (single) volume). Member is limited only by the size of PDS, number of members are limited by directory size. Huge directory provides really poor performance.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Barbara Nitz]

Last time I looked at PDSE performance was before PDSE V2. And I checked specifically PDSEs with RECFM=VB. About 10000 member in a 18000cyl PDSE library. ISPF 3.4 took about 90 seconds. That time was spent on I/O (reponse time about 4ms for each I/O). During that time about 10000 I/Os were done to that data set. AFAIK, the 'directory' of a PDSE is strewn in with the actual data (PDSEs used to use Media Manager Code for I/O, i.e. 4K blocks for each I/O). Adding the response time for the I/O more or less resulted in the 90 seconds wait time.

I believe that inserting an entry is just finding the place where it belongs and adjusting the previous and next pointer(s).

As for HFS - in a former life we ran Lotus Notes on z/OS. Over time, performance in access to the HFS degraded. IBM recommended migrating to zFS. Bad Move. We ran into several zFS software problems and ended up going back to HFS. Lo and behold, performance was MUCH better on HFS then, probably because the migration back and forth had reorganized the underlying data structures.

As for caching: The SMSPDSE1 address space used to cache the 4K blocks. Which did not help at all, because back then the maximum cache available to SMSPDSE1 was 16GB (I believe). We had about 10 of those large VB PDSE's, and together they were much bigger than the available cache, and SMSPDSE1 would cache the full 4K (which includes data). The nature of access was 'go search them all for a listing that fits', so it regularly took a long time since cache content in SMSPDSE1 got replaced. In production we ended up converting back to PDS because that was MUCH faster.

I have no idea how PDSE V2 compares to my experience. If I ever find the time I might test on a VB PDSE.

Barbara

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Barbara Nitz
I believe that inserting an entry is just finding the place where it belongs and adjusting the previous and next pointer(s).

IOW you believe its a linear linked list rather than such as a B-tree?

Ouch.

Are the directories for PDSEv1, PDSEv2, HFS, and ZFS similarly organized?

Probably NDA.

Post by Barbara Nitz
As for caching: The SMSPDSE1 address space used to cache the 4K blocks. Which did not help at all, because back then the maximum cache available to SMSPDSE1 was 16GB (I believe). We had about 10 of those large VB PDSE's, and together they were much bigger than the available cache, and SMSPDSE1 would cache the full 4K (which includes data). ...

You suggest directory blocks contain data? That would seem to optimize space
utilization at the expense of performance. And gain little except for quite small
members.

And if I NOTE at the millionth record of the thousandth member of a VB PDSE and
later POINT to it, I wonder what processing occurs?

Probably NDA.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Anthony Thompson]

There was a Redbook entitled 'Partitioned Data Set Extended Usage Guide', document number SG24-6106-01, dated May 2005 (so it very much pre-dated PDSE V2).

That Redbook stated that that PDSE directories entries were organised in a balanced B-tree, and that the PDSE pages were either directory pages, or data pages, not mixed. That information may no longer be accurate, of course.

I say was, as I searched on the Redbooks site in an effort to provide a link, using both the document title and number as search terms, but didn't find it.

Ant.

-----Original Message-----

Subject: Re: Maximum size of a PDSE library?

Post by Barbara Nitz
I believe that inserting an entry is just finding the place where it belongs and adjusting the previous and next pointer(s).

IOW you believe its a linear linked list rather than such as a B-tree?

Ouch.

Are the directories for PDSEv1, PDSEv2, HFS, and ZFS similarly organized?

Probably NDA.

Post by Barbara Nitz
As for caching: The SMSPDSE1 address space used to cache the 4K blocks. Which did not help at all, because back then the maximum cache available to SMSPDSE1 was 16GB (I believe). We had about 10 of those large VB PDSE's, and together they were much bigger than the available cache, and SMSPDSE1 would cache the full 4K (which includes data). ...

You suggest directory blocks contain data? That would seem to optimize space utilization at the expense of performance. And gain little except for quite small members.

And if I NOTE at the millionth record of the thousandth member of a VB PDSE and later POINT to it, I wonder what processing occurs?

Probably NDA.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Tony Harminc]

On 21 February 2017 at 19:51, Anthony Thompson

Post by Anthony Thompson
There was a Redbook entitled 'Partitioned Data Set Extended Usage Guide', document number SG24-6106-01, dated May 2005 (so it very much pre-dated PDSE V2).
That Redbook stated that that PDSE directories entries were organised in a balanced B-tree, and that the PDSE pages were either directory pages, or data pages, not mixed. That information may no longer be accurate, of course.
I say was, as I searched on the Redbooks site in an effort to provide a link, using both the document title and number as search terms, but didn't find it.

It's still there (as of 2017-02-21 20:31 GMT-5) as a viewable online
book, but the PDF does seem to have disappeared. Get it while you can,
I guess...

https://www.redbooks.ibm.com/redbooks/SG246106/wwhelp/wwhimpl/js/html/wwhelp.htm

Tony H.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Farley, Peter x23353]

Many thanks for that Redbook link, it had exactly what I needed in section 2.6.

Peter

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Tony Harminc
Sent: Tuesday, February 21, 2017 9:08 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Maximum size of a PDSE library?

Post by Anthony Thompson
There was a Redbook entitled 'Partitioned Data Set Extended Usage Guide', document number SG24-6106-01, dated May 2005 (so it very much pre-dated PDSE V2).
That Redbook stated that that PDSE directories entries were organised in a balanced B-tree, and that the PDSE pages were either directory pages, or data pages, not mixed. That information may no longer be accurate, of course.
I say was, as I searched on the Redbooks site in an effort to provide a link, using both the document title and number as search terms, but didn't find it.

It's still there (as of 2017-02-21 20:31 GMT-5) as a viewable online book, but the PDF does seem to have disappeared. Get it while you can, I guess...

https://www.redbooks.ibm.com/redbooks/SG246106/wwhelp/wwhimpl/js/html/wwhelp.htm

Tony H.

----------------------------------------------------------------------


This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN