Discussion:
PIPElines vs. *nix
Add Reply
Hobart Spitz
2018-06-13 13:18:52 UTC
Reply
Permalink
Raw Message
Cross posted to CMSTSO Pipelines and IBM-MAIN

Someone shared with me a performance comparison between Pipelines vs.
native *nix commands, both on OPENVM.

Under the OPENVM shell, this command ran 75 secs. with a 170M file in BFS:

$ time cat m170file.data | wc -b


This command, also under OPENVM shell with the same file, ran 9 secs.:

$ cms 'PIPE < /home/....../m170file.data | count bytes | cons'

Unfortunately, the person who sent this to me wasn't in a position to spent
any more time or resources on this, so I invite any one inclined to run a
similar comparison and post the results.

You may need something like this to avoid an abend depending on your system:

$ cms 'pipe filedesc 0 | count bytes | cons' < m170.data


Under OPENMVS, e.g., try something like:

$ tso 'PIPE < /home/....../m170file.data | count bytes | cons

(Caution, I have not used OPENMVS/USS, so the syntax could be wrong.)

MVSers who don't have PIPElines, and VMers who want to, can try comparing
*nix equivalents to REXX using LINEIN() if you have it, or EXECIO * if you
don't. This will tell us if Pipelines' design is a bigger contributor to
efficiency, or if it is the superiority of record-orientation vs.
character-orientation.

I recommend using RITA to get stage level statistics. I suspect that
scanning for CR/LF is more expensive than counting bytes in PIPElines,
while the cost might be similar in *nix.

Some variations you may want to try:

- Count lines and/or words.
- Try different mainframe *nix version.
- Add more filters.
- Add filters that drop and/or add records.
- Add some filters that change records.
- Use file(s) already in the CMS/MVS file systems. The <bfs stage has
to find the CR/LF before it can emit its output record. Having the data in
a record oriented file system avoids that overhead.
- Even though the exact *nix equivalent is difficult, something
milti-steam, e.g. with FANOUT.
- Binary data files.

Why might one care? Since what typically takes multiple *nix command lines
to accomplish typically only takes 1 command in Pipelines, making Pipelines
not only superior in development productivity vs. *nix, but it may also be
about an order of magnitude more efficient.

Thanks in advance.

- Hobart

OREXXMan
JCL is the buggy whip of 21st century computing. Stabilize it.
Put Pipelines in the z/OS base. Would you rather process data one
character at a time (Unix/C style), or one record at a time?
IBM has been looking for an HLL for program products; REXX as the C of
mainframes is that language.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Paul Gilmartin
2018-06-13 14:07:37 UTC
Reply
Permalink
Raw Message
Post by Hobart Spitz
Cross posted to CMSTSO Pipelines and IBM-MAIN
Someone shared with me a performance comparison between Pipelines vs.
native *nix commands, both on OPENVM.
$ time cat m170file.data | wc -b
$ time wc -b m170file.data
In fact, you were timing "cat", not "wc".
Post by Hobart Spitz
$ cms 'PIPE < /home/....../m170file.data | count bytes | cons'
Won't Pipelines respect a path relative to current working directory?
If not, shame on Pipelines.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Hobart Spitz
2018-06-13 23:00:30 UTC
Reply
Permalink
Raw Message
Gil, I know you mean well.

Your first two points are addressed by one answer: The goal in the test is
to compare the efficiency of record-oriented piping versus
character-by-character piping.

In the first case, bypassing the *nix piping mechanism defeats the goal of
the test.

In the second case, cat can't finish until wc starts reading the last 4k
(or so) characters at the very end of the data. With 170 M of data, that
means that the timing would be off by very much less than 0.01%, which
greatly exceeds normal variation. Further, the timings were reported to me
in more detail than I believe time reports. Therefore, it was probably not
the time output that gave 75 secs. I just choose to use the only total
time that was reported to me.

Finally, I got confused and assumed that the original PIPElines case was
abbreviated or there was a typo. I incorrectly added the $ cms and
quotes. That is entirely on me. It should have just read:

PIPE < /home/....../m170file.data | count bytes | cons


I've heard of PIPElines with 10K-15K (mostly generated) stages. Depending
on the topology, such a thing would be impossibly slow with
character-by-character piping. And that's assuming you could even do
deterministic multi-stream piping the way PIPElines does.

Think of the difference between *nix piping and PIPElines piping this way:
Your have a dinner party with a dozen guests.

Under *nix piping, you pass each and every string bean, corn kernel, and
pea to the plate of the person next to you, one-at-a-time, and only one
vegetable can move at a time. If you even finished such a dinner, you
would never get anyone to show up at another. And we haven't even
considered the cache-flooding implications.

Under PIPElines piping, whole plates of vegetables get passed around in the
familiar way. The serving utensil the "cache loader". Vegetables that you
don't want just stay in the serving bowl (slow memory or disk).

Bon appetit. :-)

I hope this helps.

OREXXMan
JCL is the buggy whip of 21st century computing. Stabilize it.
Put Pipelines in the z/OS base. Would you rather process data one
character at a time (Unix/C style), or one record at a time?
IBM has been looking for an HLL for program products; REXX is that language.

On Wed, Jun 13, 2018 at 10:07 AM, Paul Gilmartin <
Post by Paul Gilmartin
Post by Hobart Spitz
Cross posted to CMSTSO Pipelines and IBM-MAIN
Someone shared with me a performance comparison between Pipelines vs.
native *nix commands, both on OPENVM.
Under the OPENVM shell, this command ran 75 secs. with a 170M file in
$ time cat m170file.data | wc -b
$ time wc -b m170file.data
In fact, you were timing "cat", not "wc".
Post by Hobart Spitz
$ cms 'PIPE < /home/....../m170file.data | count bytes | cons'
Won't Pipelines respect a path relative to current working directory?
If not, shame on Pipelines.
-- gil
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Paul Gilmartin
2018-06-14 01:48:36 UTC
Reply
Permalink
Raw Message
Post by Hobart Spitz
Gil, I know you mean well.
Your first two points are addressed by one answer: The goal in the test is
to compare the efficiency of record-oriented piping versus
character-by-character piping.
Sorry; I missed that emphasis. COUNT WORDS is probably a fairer comparison
to "wc", since both must inspect every character.
Post by Hobart Spitz
In the first case, bypassing the *nix piping mechanism defeats the goal of
the test.
But you could amplify the sensitivity with:
cat m170file.data | cat | cat | ... | cat >/dev/null

Or Pipe into HOLE to eliminate the overhead of COUNT or wc.
Also see whether the "-u" option for "cat" makes a difference.
Is there a Pipelines stage comparably neutral to "cat" you could use?
Post by Hobart Spitz
In the second case, cat can't finish until wc starts reading the last 4k
(or so) characters at the very end of the data. With 170 M of data, that
means that the timing would be off by very much less than 0.01%, which
greatly exceeds normal variation.
"exceeds"? "is exceeded by"
I believe the z/OS UNIX kernel buffers are 131k, still an inconsequential
fraction of 170 M.
Post by Hobart Spitz
I've heard of PIPElines with 10K-15K (mostly generated) stages. Depending
on the topology, such a thing would be impossibly slow with
character-by-character piping. And that's assuming you could even do
deterministic multi-stream piping the way PIPElines does.
Deterministing multi-stream is valuable, and upstream propagation of
termination is precious. Multi-streams are possible in C; impractical
in shell.
Post by Hobart Spitz
Your have a dinner party with a dozen guests.
Under *nix piping, you pass each and every string bean, corn kernel, and
pea to the plate of the person next to you, one-at-a-time, and only one
vegetable can move at a time. If you even finished such a dinner, you
would never get anyone to show up at another. And we haven't even
considered the cache-flooding implications.
Under PIPElines piping, whole plates of vegetables get passed around in the
familiar way. The serving utensil the "cache loader". Vegetables that you
don't want just stay in the serving bowl (slow memory or disk).
I don't think that's fair. I'd expect a well-designed "cat" to issue a nonblocking
read() into an application buffer, which may be smaller than the kernel buffer,
and kernel to move the data with MVCL, not an IC; STC loop.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Loading...