Bug 53 - High memory usage when using high table id
: High memory usage when using high table id
Status: NEW
Product: Slony-I
slon
: 2.0
: PC Linux
: medium normal
Assigned To: Slony Bugs List
:
:
:
  Show dependency treegraph
 
Reported: 2008-06-23 07:32 PDT by Damian Quiroga
Modified: 2010-12-15 12:33 PST (History)
1 user (show)

See Also:


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description Damian Quiroga 2008-06-23 07:32:09 PDT
When replicating a database, the highest table id used in the configuration has
a direct impact on the memory usage of the slon processes for receiver nodes.
For example replicating a table using tableid = 1 might result in slon using 1
MB. but choosing a date-like id such as 20080623 causes the (resident) memory
usage to be about 160 MB.
Comment 1 Christopher Browne 2008-06-23 08:57:40 PDT
remote_worker.c has the following comment about this:

/*
 * Remember the fully qualified table name on the fly. This
 * might have to become a hashtable someday.
 */

Yes, indeed, if you define a table with id 20080623, this establishes the size
of the array associated with wd->tab_fqname as being at least 20080623.

For now, I'll add documentation indicating this bottleneck.
Comment 2 Jan Wieck 2010-08-13 19:28:54 PDT
I've moved this one to 2.0, since it isn't critical but should be fixed.
Instead of a hash table, this could also be implemented as a btree without
causing any dependencies, which would be a requirement to consider it for
2.0.x.
Comment 3 Christopher Browne 2010-12-03 13:14:45 PST
Some further discussion has taken place on this; see
[http://wiki.postgresql.org/wiki/SlonyBrainstorming#.2353_-_high_RAM_usage_with_high_table_.23]


Possible BSD-licensed hash table implementations:
* [http://www.cl.cam.ac.uk/~cwc22/hashtable/ C Hash Table] 
* See also [https://github.com/ryantenney/chashtable ryantenney / chashtable @
GitHub] 
* Note:
[http://xen.1045712.n5.nabble.com/Legal-concerns-added-hashtable-implementation-td2485274.html
Used in Xen]
* [http://uthash.sourceforge.net/ UTHash]
* [http://burtleburtle.net/bob/c/lookup3.c lookup3] - Public Domain