Seite 2 von 2 ErsteErste 12
Ergebnis 11 bis 17 von 17

Thema: "Out of memory" in Linux

  1. #11
    GergoH ist offline Neuer Benutzer
    Registriert seit
    02.02.2011
    Beiträge
    9

    Standard

    Thank you for your help! I could finally borrow a neat server machine with 24GB Ram, with fast HDD and CPU (System x3650 M2)
    The script is running on it now for 8 hours. 3 MB/s is processed and it is slowing down. With this preformance it will need 60 more hours to finish it. It is far more better than 60 days!

    But I still want to otimize on it because it is only the first phase of the two, and maybe I won't have enough time to finish if I need to restart it for some reason.

    Do you have any adjustment suggestions? Here are my settings:

    107,108c107
    < shared_buffers = 3GB
    ---
    > shared_buffers = 32MB
    115,116c114,115
    < work_mem = 256MB
    < maintenance_work_mem = 256MB
    ---
    > #work_mem = 1MB
    > #maintenance_work_mem = 16MB
    121c120
    < max_fsm_pages = 1592000
    ---
    > max_fsm_pages = 204800
    153c152
    < fsync = off
    ---
    > #fsync = on
    163c162
    < wal_buffers = 24MB
    ---
    > #wal_buffers = 64kB
    172c171
    < checkpoint_segments = 20
    ---
    > #checkpoint_segments = 3
    174c173
    < checkpoint_completion_target = 0.9
    ---
    > #checkpoint_completion_target = 0.5
    205c204
    < random_page_cost = 1.5
    ---
    > #random_page_cost = 4.0
    209c208
    < effective_cache_size = 8GB
    ---
    > #effective_cache_size = 128MB
    383c381
    < autovacuum = off
    ---
    > #autovacuum = on


    Thanks in advance!

  2. #12
    akretschmer ist offline Super-Moderator
    Registriert seit
    29.10.2006
    Ort
    Dresden
    Beiträge
    5.468

    Standard

    Zitat Zitat von GergoH Beitrag anzeigen
    Thank you for your help! I could finally borrow a neat server machine with 24GB Ram, with fast HDD and CPU (System x3650 M2)
    The script is running on it now for 8 hours. 3 MB/s is processed and it is slowing down. With this preformance it will need 60 more hours to finish it. It is far more better than 60 days!

    But I still want to otimize on it because it is only the first phase of the two, and maybe I won't have enough time to finish if I need to restart it for some reason.

    Do you have any adjustment suggestions? Here are my settings:

    107,108c107
    < shared_buffers = 3GB
    ---
    > shared_buffers = 32MB
    115,116c114,115
    < work_mem = 256MB
    < maintenance_work_mem = 256MB
    ---
    > #work_mem = 1MB
    > #maintenance_work_mem = 16MB
    121c120
    < max_fsm_pages = 1592000
    ---
    > max_fsm_pages = 204800
    153c152
    < fsync = off
    ---
    > #fsync = on
    163c162
    < wal_buffers = 24MB
    ---
    > #wal_buffers = 64kB
    172c171
    < checkpoint_segments = 20
    ---
    > #checkpoint_segments = 3
    174c173
    < checkpoint_completion_target = 0.9
    ---
    > #checkpoint_completion_target = 0.5
    205c204
    < random_page_cost = 1.5
    ---
    > #random_page_cost = 4.0
    209c208
    < effective_cache_size = 8GB
    ---
    > #effective_cache_size = 128MB
    383c381
    < autovacuum = off
    ---
    > #autovacuum = on


    Thanks in advance!
    I think, you can increase shared_mem up to 6 GB, work_mem is maybe too high. Why fsync = off? It's dangerous! Why autovacuum off? Is it a dedicated pg-server? If yes, you can increase effective_cache_size up to 20 GB. Why random_page_cost to 1.5, i think, it's too low. Which version you are running? I'm asking because of max_fsm_pages, it's deprecated in newer versions.

    Sorry for my bad english, it's not my native language.

    Andreas
    --
    ads Das hatte dir akretschmer gestern schon gesagt und siehe da sobald du das machst, funktioniert es ;-)
    https://www.xing.com/profile/Andreas_Kretschmer7

  3. #13
    GergoH ist offline Neuer Benutzer
    Registriert seit
    02.02.2011
    Beiträge
    9

    Standard

    Thanks for your input Andreas. I'll apply your suggestions when the current run ends.

    The process runs on a pg dedicated server with version of 8.3.12. This version is the most supported by the mapping application I want to use. As for fsync and autovacuum, I'm aware of that turning them off is dangerous, but in this case these are disabled for this initial phase only to make the process even faster.

  4. #14
    GergoH ist offline Neuer Benutzer
    Registriert seit
    02.02.2011
    Beiträge
    9

    Standard

    The kernel (oom-killer) killed the postgre process again . I don't understand what I am doing wrong.
    It's a default Debian and PostgreSQL. Nothing was changed besides what I've mentioned. (in this case Debian's shmmax was set to 8GB, and the shmall accordingly)

    Now I'm setting the overcommit mode of the Debian to strict, applying the new settings to see what happens...

    Here is the debian's message log of this incident. Most of this is chinese for me, but maybe there is somebody among you who can make conclusions:


    Feb 17 12:41:57 psql-host kernel: [249339.209591] sshd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
    Feb 17 12:41:57 psql-host kernel: [249339.209596] Pid: 4315, comm: sshd Not tainted 2.6.26-2-amd64 #1
    Feb 17 12:41:57 psql-host kernel: [249339.209597]
    Feb 17 12:41:57 psql-host kernel: [249339.209598] Call Trace:
    Feb 17 12:41:57 psql-host kernel: [249339.209612] [<ffffffff80273908>] oom_kill_process+0x57/0x1dc
    Feb 17 12:41:57 psql-host kernel: [249339.209616] [<ffffffff8023b4fd>] __capable+0x9/0x1c
    Feb 17 12:41:57 psql-host kernel: [249339.209618] [<ffffffff80273c33>] badness+0x188/0x1c7
    Feb 17 12:41:57 psql-host kernel: [249339.209622] [<ffffffff80273e67>] out_of_memory+0x1f5/0x28e
    Feb 17 12:41:57 psql-host kernel: [249339.209627] [<ffffffff80276bb8>] __alloc_pages_internal+0x31d/0x3bf
    Feb 17 12:41:57 psql-host kernel: [249339.209633] [<ffffffff8027886e>] __do_page_cache_readahead+0x79/0x183
    Feb 17 12:41:57 psql-host kernel: [249339.209637] [<ffffffff8027311d>] filemap_fault+0x15d/0x33c
    Feb 17 12:41:57 psql-host kernel: [249339.209642] [<ffffffff8027e69c>] __do_fault+0x50/0x3e8
    Feb 17 12:41:57 psql-host kernel: [249339.209647] [<ffffffff80281a5f>] handle_mm_fault+0x452/0x8de
    Feb 17 12:41:57 psql-host kernel: [249339.209651] [<ffffffff802461d1>] autoremove_wake_function+0x0/0x2e
    Feb 17 12:41:57 psql-host kernel: [249339.209655] [<ffffffff80238ba1>] current_fs_time+0x1e/0x24
    Feb 17 12:41:57 psql-host kernel: [249339.209659] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
    Feb 17 12:41:57 psql-host kernel: [249339.209663] [<ffffffff8029b968>] vfs_read+0x11e/0x152
    Feb 17 12:41:57 psql-host kernel: [249339.209665] [<ffffffff8023dea3>] recalc_sigpending+0xe/0x38
    Feb 17 12:41:57 psql-host kernel: [249339.209669] [<ffffffff8042ab79>] error_exit+0x0/0x60
    Feb 17 12:41:57 psql-host kernel: [249339.209674]
    Feb 17 12:41:57 psql-host kernel: [249339.209675] Mem-info:
    Feb 17 12:41:57 psql-host kernel: [249339.209676] Node 0 DMA per-cpu:
    Feb 17 12:41:57 psql-host kernel: [249339.209679] CPU 0: hi: 0, btch: 1 usd: 0
    Feb 17 12:41:57 psql-host kernel: [249339.209681] CPU 1: hi: 0, btch: 1 usd: 0
    Feb 17 12:41:57 psql-host kernel: [249339.209682] CPU 2: hi: 0, btch: 1 usd: 0
    Feb 17 12:41:57 psql-host kernel: [249339.209684] CPU 3: hi: 0, btch: 1 usd: 0
    Feb 17 12:41:57 psql-host kernel: [249339.209685] Node 0 DMA32 per-cpu:
    Feb 17 12:41:57 psql-host kernel: [249339.209687] CPU 0: hi: 186, btch: 31 usd: 158
    Feb 17 12:41:57 psql-host kernel: [249339.209689] CPU 1: hi: 186, btch: 31 usd: 175
    Feb 17 12:41:57 psql-host kernel: [249339.209691] CPU 2: hi: 186, btch: 31 usd: 124
    Feb 17 12:41:57 psql-host kernel: [249339.209692] CPU 3: hi: 186, btch: 31 usd: 123
    Feb 17 12:41:57 psql-host kernel: [249339.209694] Node 0 Normal per-cpu:
    Feb 17 12:41:57 psql-host kernel: [249339.209696] CPU 0: hi: 186, btch: 31 usd: 177
    Feb 17 12:41:57 psql-host kernel: [249339.209697] CPU 1: hi: 186, btch: 31 usd: 149
    Feb 17 12:41:57 psql-host kernel: [249339.209699] CPU 2: hi: 186, btch: 31 usd: 185
    Feb 17 12:41:57 psql-host kernel: [249339.209700] CPU 3: hi: 186, btch: 31 usd: 167
    Feb 17 12:41:57 psql-host kernel: [249339.209703] Active:5651793 inactive:961599 dirty:0 writeback:0 unstable:0
    Feb 17 12:41:57 psql-host kernel: [249339.209705] free:32269 slab:8346 mapped:59 pagetables:23779 bounce:0
    Feb 17 12:41:57 psql-host kernel: [249339.209706] Node 0 DMA free:11600kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:10660kB pages_scanned:0 all_unreclaimable? yes
    Feb 17 12:41:57 psql-host kernel: [249339.209710] lowmem_reserve[]: 0 1965 26205 26205
    Feb 17 12:41:57 psql-host kernel: [249339.209713] Node 0 DMA32 free:98484kB min:1552kB low:1940kB high:2328kB active:829744kB inactive:693388kB present:2012496kB pages_scanned:2456458 all_unreclaimable? yes
    Feb 17 12:41:57 psql-host kernel: [249339.209717] lowmem_reserve[]: 0 0 24240 24240
    Feb 17 12:41:57 psql-host kernel: [249339.209720] Node 0 Normal free:18992kB min:19160kB low:23948kB high:28740kB active:21777172kB inactive:3153264kB present:24821760kB pages_scanned:45436605 all_unreclaimable? yes
    Feb 17 12:41:57 psql-host kernel: [249339.209724] lowmem_reserve[]: 0 0 0 0
    Feb 17 12:41:57 psql-host kernel: [249339.209726] Node 0 DMA: 2*4kB 3*8kB 3*16kB 4*32kB 2*64kB 0*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11600kB
    Feb 17 12:41:57 psql-host kernel: [249339.209733] Node 0 DMA32: 45*4kB 54*8kB 25*16kB 18*32kB 26*64kB 10*128kB 13*256kB 11*512kB 11*1024kB 28*2048kB 4*4096kB = 98484kB
    Feb 17 12:41:57 psql-host kernel: [249339.209740] Node 0 Normal: 85*4kB 2*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 4*4096kB = 18964kB
    Feb 17 12:41:57 psql-host kernel: [249339.209747] 695604 total pagecache pages
    Feb 17 12:41:57 psql-host kernel: [249339.209748] Swap cache: add 7431531, delete 7431528, find 738559/1270117
    Feb 17 12:41:57 psql-host kernel: [249339.209750] Free swap = 0kB
    Feb 17 12:41:57 psql-host kernel: [249339.209751] Total swap = 11590856kB
    Feb 17 12:41:57 psql-host kernel: [249339.295405] 6815744 pages of RAM
    Feb 17 12:41:57 psql-host kernel: [249339.295407] 113880 reserved pages
    Feb 17 12:41:57 psql-host kernel: [249339.295409] 906 pages shared
    Feb 17 12:41:57 psql-host kernel: [249339.295410] 3 pages swap cached

  5. #15
    GergoH ist offline Neuer Benutzer
    Registriert seit
    02.02.2011
    Beiträge
    9

    Standard

    I was thinking, and what came to my mind is that PostGIS is not the part of the default pg. So on google I've found that postgis versions prior to 1.3.5 have some memory leak issues. As I have 1.3.3 installed, I upgrade it to the latest, hoping that it solves my case.

    I'll inform you as soon as I get some results.

  6. #16
    GergoH ist offline Neuer Benutzer
    Registriert seit
    02.02.2011
    Beiträge
    9

    Standard

    It was not the PostGIS.


    WARNING: terminating connection because of crash of another server process
    DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

  7. #17
    ads
    ads ist offline Super-Moderator
    Registriert seit
    08.02.2006
    Ort
    MD
    Beiträge
    4.056

    Standard

    Take a look into your syslog.
    Xandrian: Kurz gesagt: Du hast ein falsches Konzept und willst nicht auf ads hören

    Jetzt NEU: PostgreSQL - Datenbankpraxis für Anwender, Administratoren und Entwickler

    PostgreSQL Service & Support

Seite 2 von 2 ErsteErste 12

Ähnliche Themen

  1. Antworten: 7
    Letzter Beitrag: 03.02.2012, 09:58
  2. Antworten: 1
    Letzter Beitrag: 28.08.2009, 13:51
  3. Shared-memory (linux) shmmax?
    Von CenText im Forum Konfiguration
    Antworten: 4
    Letzter Beitrag: 18.01.2008, 20:52
  4. "Group-by" und "max" Funktionen verknüpfen
    Von superguddy777 im Forum SQL
    Antworten: 1
    Letzter Beitrag: 22.08.2007, 08:35
  5. OLD.sum(...) ->record "old" has no field "sum"
    Von Mster im Forum Trigger und Funktionen
    Antworten: 5
    Letzter Beitrag: 18.01.2007, 15:00

Berechtigungen

  • Neue Themen erstellen: Nein
  • Themen beantworten: Nein
  • Anhänge hochladen: Nein
  • Beiträge bearbeiten: Nein
  •  

Links: Tobias Bauer - IT- und Funkdienstleistungen   -   Bilderrahmen-kaufen.de - Der Onlineshop für Bilderrahmen & Fotorahmen   -   Funkforum