trouble with nested operators

If we don't know about 'em, we can't fix 'em.

trouble with nested operators

PostPosted by drab » Sun Feb 26, 2017 7:35 am

1J2 3J4 ∘.(○⍨) 9 11
SYMBOL TABLE FULL

⎕sysver
0.5.1.31 Sat Feb 25 13:24:50 2017 Win/64
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by forummaster » Mon Feb 27, 2017 1:42 am

drab wrote:1J2 3J4 ∘.(○⍨) 9 11
SYMBOL TABLE FULL

⎕sysver
0.5.1.31 Sat Feb 25 13:24:50 2017 Win/64

Thanks for the bug report. I'm unable to duplicate it, but this kind of error message reflects a real situation where you've overwhelmed the symbol table with a large number of small constants -- any simple scalar integer, character, or FLT will do. You can override the default size with a command line parameter "-symtabsize=NNN" where NNN is in units of 1K. The default sizes for the Symbol Table and Hash Table may be found from the menu "Help | About".
forummaster
 
Posts: 545
Joined: Wed Jan 23, 2013 1:00 pm

Re: trouble with nested operators

PostPosted by drab » Mon Feb 27, 2017 7:01 am

I did not have a lot of small constants, but I did have some large arrays of floats and complex. Maybe somehow one of them became mixed.

In any case, I can no longer duplicate that problem now either.

Is there any way to report the current usage of the symbol table?
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by forummaster » Mon Feb 27, 2017 7:42 am

drab wrote:I did not have a lot of small constants, but I did have some large arrays of floats and complex. Maybe somehow one of them became mixed.

In any case, I can no longer duplicate that problem now either.

Is there any way to report the current usage of the symbol table?

A nested array of numbers could contain Symbol Table Entries if you had any simple scalars in the array. That is, the array 1 2.5 3 (4 5) would use three STEs (one each) for the 1 2.5 3. Nested arrays are stored as an array of ptrs where the 1 2.5 3 in this case are saved in the array each as ptrs to a STE. The good news is that STEs are reused so that there is only one STE for every (say) 2.5 as an STE, but then there are a lot of possible numeric constants. I decided to go this route so as not to take up storage in global memory for small constants. Homogeneous arrays store their data in memory without the need for ptrs to STEs. Only nested or heterogeneous arrays resort to this artifice.

Reporting the current STE usage as well as that of the Hash Table would be useful.

I suppose I could change the size of the Symbol Table as well as the Hash Table on the fly by trundling through every nested or heterogeneous array and rebasing the STEs and HTEs. This could be done only with an empty State Indicator. My ToDo list just got longer. )SYMB here I come.
forummaster
 
Posts: 545
Joined: Wed Jan 23, 2013 1:00 pm

Re: trouble with nested operators

PostPosted by drab » Mon Feb 27, 2017 8:48 am

I definitely had some large nested arrays with different data types in them.
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by drab » Sun Mar 05, 2017 11:43 am

I still sometimes get SYMBOL TABLE FULL for unknown reasons.

The most recent one was on a 2 ⎕dr¨ of an array that wasn't even very big.

I was able to clear that one up by doing save,clear,copy.
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by drab » Sun Mar 05, 2017 12:26 pm

I found a simple expression that causes SYMBOL TABLE FULL in a clear ws.

v←'x',⍳1e6
SYMBOL TABLE FULL

I know that large mixed arrays can be problematic in this area, but that limit is way too small.
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by forummaster » Mon Mar 06, 2017 1:25 am

drab wrote:I found a simple expression that causes SYMBOL TABLE FULL in a clear ws.

v←'x',⍳1e6
SYMBOL TABLE FULL

I know that large mixed arrays can be problematic in this area, but that limit is way too small.

You are creating 1,000,001 STEs and that's "way too small"?

The latest Alpha version implements a preliminary version of )SYMB. It'll tell you the maximum size to which the Sym & Hsh tables can grow, the total (i.e. current) size, and the number of entries in use.

You might still need to )SAVE/)CLEAR/COPY or use the command line argument -symtabsize=NNN. The number NNN is 256 by default and is in units of 1K. You might also need -hshtabsize=NNN which has the same default and units.
forummaster
 
Posts: 545
Joined: Wed Jan 23, 2013 1:00 pm

Re: trouble with nested operators

PostPosted by drab » Mon Mar 06, 2017 7:43 am

Yes, I think it's too small.

You done a great job of pushing out most of the limits so that they are actually hard to hit.
That's fantastic.
But this one's too easy to hit.

I know it might be a very big job to dynamically expand the symbol table, and that's probably and not worth the effort.
But it could be very worthwhile to at least be able to set a higher limit in a clear workspace.

I like the new )symb command, but please explain it further.
Specifically, what is the difference between Total and Inuse?
And how is the Symtab different from the Hshtab?

How big is each entry in the symbol table? I'm guessing somewhere around 16 bytes or so. Maybe more for long names.
That would make the total size of the symbol table about 4Mb for the default Max value of 262144. Is that close?
But 4Mb is not so much to worry about these days, so a significant increase in the maximum would be quite tolerable, at least for me, even if my guess is off by quite a bit.
drab
 
Posts: 289
Joined: Thu Oct 09, 2014 6:23 am

Re: trouble with nested operators

PostPosted by forummaster » Sat Mar 11, 2017 8:33 pm

drab wrote:I know it might be a very big job to dynamically expand the symbol table, and that's probably and not worth the effort.
But it could be very worthwhile to at least be able to set a higher limit in a clear workspace.

I like the new )symb command, but please explain it further.
Specifically, what is the difference between Total and Inuse?
And how is the Symtab different from the Hshtab?

How big is each entry in the symbol table? I'm guessing somewhere around 16 bytes or so. Maybe more for long names.
That would make the total size of the symbol table about 4Mb for the default Max value of 262144. Is that close?
But 4Mb is not so much to worry about these days, so a significant increase in the maximum would be quite tolerable, at least for me, even if my guess is off by quite a bit.

It's not trivial to expand the Symbol and Hash Tables, but it's worthwhile. Of course it can be done with a clear SI only. I'm working on it slowly as there are several places where a STE/HTE can be tucked away and I have to find them all.

The )SYMB command is preliminary. In a CLEAR WS, it looks like this

SymTab:  Max =   262144, Total =     4274, Inuse = 176
HshTab: Max = 2097152, Total = 131072, Inuse = 176

* The Max value is the maximum # entries (STEs or HTEs) to which the table can be expanded. The allocation method used is VirtualAlloc (uncommitted memory) which allocates address space without having to back it up with RAM. This method allows one to allocate a large address space without penalizing users with low available RAM. The initial value is fixed, but may be changed via the command line switches -symtabsize=NNN and -hshtabsize=NNN. The purpose of the )SYMB command is to resize those tables dynamically in any workspace with a clear SI without exiting the interpreter and restarting with a new command line.
* The Total value (not a good name, I'll change it) is the current # entries in the Table. It represents the entries backed up by RAM.
* The Inuse value is the # entries currently with an assigned value. These represent values such as 0, 1, and ' ' and various other constants initialized at startup.

I use a linear hashing technique I found somewhere: Witold Litwin, "Linear hashing: A new tool for file and table addressing", Proc. 6th Conference on Very Large Databases, pages 212-223, 1980. This technique allows the Hash table to grow linearly(!) and dynamically. It's very clever. It is used when the Inuse value reaches the Total value of the Hash table. At this time, more RAM is allocated to back up the new addresses. When the Total value reaches the Max value, you get a SYMBOL or HASH TABLE FULL error message.

The byte sizes for STEs and HTEs differ based upon whether the system is 32- or 64-bit due to pointer sizes (4 or 8 bytes):
      32-bit   64-bit
STE 32 44
HTE 28 44

I hope this helps.
forummaster
 
Posts: 545
Joined: Wed Jan 23, 2013 1:00 pm

Next

Return to Bug Reports

Who is online

Users browsing this forum: No registered users and 3 guests

cron