[Home]

Summary:DAHLIN-00013: No room for chans on E1 span (insmod: page allocation failure. order:5)
Reporter:Tzafrir Cohen (tzafrir)Labels:
Date Opened:2008-05-15 07:52:14Date Closed:2008-07-02 12:22:48
Priority:MinorRegression?No
Status:Closed/CompleteComponents:General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:An E1 span requires a span of at least 31 channels. All the channels of a span are allocated in a single array. struct zt_chan takes around 2200 bytes. Thus an array of 31 of them takes around 68200 . This is more than 16 pages (4096kB), and hence the allocation requires a free block of 32 pages - order 5.

If the system is low enough on memory and/or fragmented enough, there will not be such a block available.

In such a case we get a scary-looking oops-like message, though in practice we simply have a failed memory allocation that seems to be handled gracefully by the driver.

How do we get rid of this big block of more than 64kB of memory? Maybe if we manage to cut some 100 bytes or so from that struct ithat allocation will become an order 4 (16 pages) one and thus a bit less of a problem.

May 15 08:37:53 brodni kernel: TE2XXP version c01a016a, burst ON
May 15 08:37:53 brodni kernel: insmod: page allocation failure. order:5, mode:0xd0
May 15 08:37:53 brodni kernel:  [<c04540ed>] __alloc_pages+0x26a/0x27e
May 15 08:37:53 brodni kernel:  [<c0467138>] cache_alloc_refill+0x26d/0x44f
May 15 08:37:53 brodni kernel:  [<c0466ec1>] kmem_cache_alloc+0x41/0x4b
May 15 08:37:53 brodni kernel:  [<cfbf6f1d>] t4_init_one+0x533/0xea8 [wct4xxp]
May 15 08:37:53 brodni kernel:  [<c05439eb>] __driver_attach+0x0/0x6b
May 15 08:37:53 brodni kernel:  [<c04e6bf4>] pci_device_probe+0x36/0x57
May 15 08:37:53 brodni kernel:  [<c0543945>] driver_probe_device+0x42/0x8b
May 15 08:37:53 brodni kernel:  [<c0543a2f>] __driver_attach+0x44/0x6b
May 15 08:37:53 brodni kernel:  [<c054344a>] bus_for_each_dev+0x37/0x59
May 15 08:37:53 brodni kernel:  [<c05438af>] driver_attach+0x11/0x13
May 15 08:37:53 brodni kernel:  [<c05439eb>] __driver_attach+0x0/0x6b
May 15 08:37:53 brodni kernel:  [<c0543152>] bus_add_driver+0x64/0xfd
May 15 08:37:53 brodni kernel:  [<c04e6d22>] __pci_register_driver+0x47/0x63
May 15 08:37:53 brodni kernel:  [<cf83300f>] t4_init+0xf/0x1a [wct4xxp]
May 15 08:37:53 brodni kernel:  [<c043d0d4>] sys_init_module+0x16e7/0x186a
May 15 08:37:53 brodni kernel:  [<c046a5df>] do_sync_read+0xb6/0xf1
May 15 08:37:53 brodni kernel:  [<c0403eff>] syscall_call+0x7/0xb
May 15 08:37:53 brodni kernel:  =======================
May 15 08:37:53 brodni kernel: Mem-info:
May 15 08:37:53 brodni kernel: DMA per-cpu:
May 15 08:37:53 brodni kernel: cpu 0 hot: high 0, batch 1 used:0
May 15 08:37:53 brodni kernel: cpu 0 cold: high 0, batch 1 used:0
May 15 08:37:53 brodni kernel: DMA32 per-cpu: empty
May 15 08:37:53 brodni kernel: Normal per-cpu:
May 15 08:37:53 brodni kernel: cpu 0 hot: high 90, batch 15 used:77
May 15 08:37:53 brodni kernel: cpu 0 cold: high 30, batch 7 used:27
May 15 08:37:53 brodni kernel: HighMem per-cpu: empty
May 15 08:37:53 brodni kernel: Free pages:       79676kB (0kB HighMem)
May 15 08:37:53 brodni kernel: Active:27735 inactive:3608 dirty:81 writeback:0 unstable:0 free:19919 slab:6728 mapped:4772 pagetables:716
May 15 08:37:53 brodni kernel: DMA free:8020kB min:132kB low:164kB high:196kB active:140kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no
May 15 08:37:53 brodni kernel: lowmem_reserve[]: 0 0 223 223
May 15 08:37:53 brodni kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 15 08:37:54 brodni kernel: lowmem_reserve[]: 0 0 223 223
May 15 08:37:54 brodni kernel: Normal free:71656kB min:1844kB low:2304kB high:27
64kB active:110800kB inactive:14432kB present:229312kB pages_scanned:153 all_unreclaimable? no
May 15 08:37:54 brodni kernel: lowmem_reserve[]: 0 0 0 0
May 15 08:37:54 brodni kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 15 08:37:54 brodni kernel: lowmem_reserve[]: 0 0 0 0
May 15 08:37:54 brodni kernel: DMA: 177*4kB 156*8kB 119*16kB 64*32kB 31*64kB 1*1
28kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8020kB
May 15 08:37:54 brodni kernel: DMA32: empty
May 15 08:37:54 brodni kernel: Normal: 194*4kB 4050*8kB 1689*16kB 304*32kB 25*64
kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 71656kB
May 15 08:37:54 brodni kernel: Swap cache: add 1028, delete 1028, find 805/878, race 0+0
Comments:By: Kevin P. Fleming (kpfleming) 2008-07-01 13:31:02

If the system has so little memory available that kmalloc() of 128KB is failing, I don't see how any application that services an E1 could reasonably be expected to run on said system :-)

If we decide to change this to single allocations, there will be a very slight performance penalty due to having to access the zt_chan structures via pointer instead of just indexing an array, but that negligible. Feel free to provide a patch to improve this situation if you like, but keep in mind that if we do it we'll have to do it for *all* the digital span drivers to be consistent.
By: Tzafrir Cohen (tzafrir) 2008-07-01 14:05:00

The problem here is not to find a some 128kb, but to find an available memory block at that size.
After various allocations/deallocations the kernel memory gets fragmented, and the only way out is a reboot.

We often run into this with system of 256MB after enough loads/unloads of E1 devices.
By: Digium Subversion (svnbot) 2008-07-02 08:24:24

Repository: dahdi
Revision: 4513

U   linux/trunk/drivers/dahdi/dahdi-base.c
U   linux/trunk/drivers/dahdi/dahdi_dynamic.c
U   linux/trunk/drivers/dahdi/tor2.c
U   linux/trunk/drivers/dahdi/wct1xxp.c
U   linux/trunk/drivers/dahdi/wct4xxp/base.c
U   linux/trunk/drivers/dahdi/wcte11xp.c
U   linux/trunk/drivers/dahdi/wcte12xp/base.c
U   linux/trunk/drivers/dahdi/wcte12xp/vpmadt032.c
U   linux/trunk/drivers/dahdi/wcte12xp/wcte12xp.h
U   linux/trunk/drivers/dahdi/xpp/card_fxo.c
U   linux/trunk/drivers/dahdi/xpp/card_fxs.c
U   linux/trunk/drivers/dahdi/xpp/card_pri.c
U   linux/trunk/drivers/dahdi/xpp/xbus-pcm.c
U   linux/trunk/drivers/dahdi/xpp/xpd.h
U   linux/trunk/drivers/dahdi/xpp/xpp_dahdi.c
U   linux/trunk/include/dahdi/kernel.h

------------------------------------------------------------------------
r4513 | kpfleming | 2008-07-02 08:24:22 -0500 (Wed, 02 Jul 2008) | 5 lines

break up large memory allocations made by digital span drivers into smaller ones (one allocation for each dahdi_chan structure, separate from any private structures used by the driver)

(closes issue DAHLIN-13)
Reported by: tzafrir

------------------------------------------------------------------------

http://svn.digium.com/view/dahdi?view=rev&revision=4513
By: Digium Subversion (svnbot) 2008-07-02 12:22:18

Repository: dahdi
Revision: 4514

U   linux/trunk/drivers/dahdi/dahdi_dummy.c
U   linux/trunk/drivers/dahdi/dahdi_dynamic.c
U   linux/trunk/drivers/dahdi/wcfxo.c
U   linux/trunk/drivers/dahdi/wctdm.c
U   linux/trunk/drivers/dahdi/wctdm24xxp/base.c
U   linux/trunk/drivers/dahdi/wctdm24xxp/wctdm24xxp.h

------------------------------------------------------------------------
r4514 | kpfleming | 2008-07-02 09:05:57 -0500 (Wed, 02 Jul 2008) | 4 lines

fix remaining drivers broken by pointer type change in 'struct dahdi_span'... wctdm24xxp still broken, will fix in a couple of hours

(related to issue DAHLIN-13)

------------------------------------------------------------------------

http://svn.digium.com/view/dahdi?view=rev&revision=4514
By: Digium Subversion (svnbot) 2008-07-02 12:22:48

Repository: dahdi
Revision: 4521

U   linux/trunk/drivers/dahdi/pciradio.c
U   linux/trunk/drivers/dahdi/wctdm24xxp/base.c

------------------------------------------------------------------------
r4521 | kpfleming | 2008-07-02 12:21:44 -0500 (Wed, 02 Jul 2008) | 4 lines

fix up the last few drivers for the new single-allocation channel changes

(related to issue DAHLIN-13)

------------------------------------------------------------------------

http://svn.digium.com/view/dahdi?view=rev&revision=4521