Paging-space problems

[Alex Martelli]

rogers at rogers.austin.ibm.com (Mark D. Rogers/100000) writes:
	...
[after a clear explanation of what AIX is doing re malloc()]
>We have at Least two very distinct classes of customers where
>paging space allocation is concerned:
>1.  Those such as yourselves, who want your application to either
>    run, or not, based upon how much backing storage you have.
>    If your app. doesn't run, go buy another disk & add paging space.
>2.  Customers who like to allocate all the virtual memory they can,
>    knowing that it will never all be used. This, alleviates the
>    need for any complicated run-time memory management schemes in
>    their model. It is a very convenient programming construct.

Thanks for the explanation.  However, in our case it's not really
that the application "would not run" - it would run fine as long
as the user was only doing solid models whose complexity were
compatible with his/her amount of paging space, and would give
a harmless warning if a model turned out to be too complex for
that (and in many cases there will be some work-around for such
resource limitations).
I would assume that many applications where the user interactively
asks for memory-consuming operations, from symbolic maths to
statistical data analysis, would be similar to solid modeling in
this respect.  Pity that this class of apps has been sacrificed
to others who apparently need garbage-collection stuff and appear
to be too lazy to do it:-).  
Another consideration is that many applications will be PORTED from
the huge existing Unix base to AIX; these will NOT expect unreliable,
over-committed malloc()!  For newly developed apps it may be a draw,
but considering potential portings to AIX, I believe the 2 vs 1
choice was inferior.  Pity!

Touching all pages and calling psdanger() on EACH malloc() is
WAY too much overhead, I think.  I believe I will have to rewrite a
malloc()/free()/realloc() package, putting all overhead only on the
sbrk(); it's either that, or have to explain to customers how and why
our solid modeler is so fragile on AIX, while it's solid as a rock
on HP/UX, Ultrix, Sun/Os, SONY/NeWS, and so on and on!

>From a historical perspective (for your information) we had
>a number of `type 2' customers Early on on the RT, and
>that influenced the System V-like behaviour of the RT somewhat.

>It also has something to do with why we, on the RISC System/6000
>still want to allow large virtual memory objects efficiently.

>Basically what we have on the RISC System/6000, is a hybrid attempt
>at allowing both types of customers. SIGDANGER is a compromise attempt
>to allow them to co-exist. Admittedly, it is not perfect, however,
>no matter what route one goes in, on this issue, we have found
>thorns in the path. We are continuing to investigate the entire
>issue, and welcome any comments.

I hope this input is some use!  If you would just include, say,
a libmalloc.a which does all necessary checks on sbrk(), for
"compatibility with non-overcommitting Unices", it would be nice.
Best, I believe, would be a tunable parameter to force sbrk() to
non-overcommittance on a system-wide basis; I don't really see how
you could make SOME processes overcommit while all defaults to
safe allocation, or viceversa, but if you could, that would
definitely be a jump upwards in quality, to go with others that
AIX has undoubtedly.

Feel free to follow this up either here or by email (but in this
case to staff at cadlab.sublink.org, please - it's not really a
"personal interest" thing, although I'm replying from home - I
can't really afford a RS/6000 machine at home...:-).  And thanks
again for your clear explanation and comments.
-- 
Alex Martelli - (home snailmail:) v. Barontini 27, 40138 Bologna, ITALIA
Email: (work:) staff at cadlab.sublink.org, (home:) alex at am.sublink.org
Phone: (work:) ++39 (51) 371099, (home:) ++39 (51) 250434; 
Fax: ++39 (51) 366964 (work only), Fidonet: 332/401.3 (home only).