Bug 53 - High memory usage when using high table id
Summary: High memory usage when using high table id
Status: NEW
Alias: None
Product: Slony-I
Classification: Unclassified
Component: slon (show other bugs)
Version: 2.0
Hardware: PC Linux
: medium normal
Assignee: Slony Bugs List
URL:
Depends on:
Blocks:
 
Reported: 2008-06-23 07:32 UTC by Damian Quiroga
Modified: 2010-12-15 12:33 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Damian Quiroga 2008-06-23 07:32:09 UTC
When replicating a database, the highest table id used in the configuration has a direct impact on the memory usage of the slon processes for receiver nodes.
For example replicating a table using tableid = 1 might result in slon using 1 MB. but choosing a date-like id such as 20080623 causes the (resident) memory usage to be about 160 MB.
Comment 1 Christopher Browne 2008-06-23 08:57:40 UTC
remote_worker.c has the following comment about this:

/*
 * Remember the fully qualified table name on the fly. This
 * might have to become a hashtable someday.
 */

Yes, indeed, if you define a table with id 20080623, this establishes the size of the array associated with wd->tab_fqname as being at least 20080623.

For now, I'll add documentation indicating this bottleneck.
Comment 2 Jan Wieck 2010-08-13 19:28:54 UTC
I've moved this one to 2.0, since it isn't critical but should be fixed. Instead of a hash table, this could also be implemented as a btree without causing any dependencies, which would be a requirement to consider it for 2.0.x.
Comment 3 Christopher Browne 2010-12-03 13:14:45 UTC
Some further discussion has taken place on this; see [http://wiki.postgresql.org/wiki/SlonyBrainstorming#.2353_-_high_RAM_usage_with_high_table_.23]


Possible BSD-licensed hash table implementations:
* [http://www.cl.cam.ac.uk/~cwc22/hashtable/ C Hash Table] 
* See also [https://github.com/ryantenney/chashtable ryantenney / chashtable @ GitHub] 
* Note: [http://xen.1045712.n5.nabble.com/Legal-concerns-added-hashtable-implementation-td2485274.html Used in Xen]
* [http://uthash.sourceforge.net/ UTHash]
* [http://burtleburtle.net/bob/c/lookup3.c lookup3] - Public Domain