Unloading unused kernel modules (NAT speed)

[Rafał Miłecki]

On 3.02.2023 21:26, Florian Fainelli wrote:
> Have you managed to get a `perf record` or `perf top` capture to see whether we might be going a little too far into iptables-raw code rather than bailing out early if we are not utilizing it?

Just did.

It doesn't hele me much but maybe you can figure out sth out of it. It
seems that with iptable_raw loaded v7_dma_inv_range() starts getting
partially called from swapper. Without iptable_raw it's called from
ksoftirqd only.

1. Fast (iptable_raw NOT loaded)
Samples: 959K of event 'cycles', Event count (approx.): 333951460472
Overhead  Command          Shared Object        Symbol
   10,39%  ksoftirqd/0      [kernel.kallsyms]    [k] v7_dma_inv_range
    5,61%  ksoftirqd/0      [kernel.kallsyms]    [k] l2c210_inv_range
    5,22%  ksoftirqd/1      [kernel.kallsyms]    [k] v7_dma_clean_range
    3,82%  ksoftirqd/1      [kernel.kallsyms]    [k] l2c210_clean_range
    2,86%  ksoftirqd/1      [kernel.kallsyms]    [k] __netif_receive_skb_core
    2,18%  ksoftirqd/1      [kernel.kallsyms]    [k] bgmac_start_xmit
    1,65%  ksoftirqd/1      [kernel.kallsyms]    [k] fib_table_lookup
    1,63%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_rcv_core
    1,45%  ksoftirqd/1      [kernel.kallsyms]    [k] __dev_queue_xmit
    1,41%  ksoftirqd/0      [kernel.kallsyms]    [k] __siphash_unaligned
    1,34%  swapper          [kernel.kallsyms]    [k] v7_dma_inv_range
    1,24%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_route_input_slow
    1,19%  ksoftirqd/0      [kernel.kallsyms]    [k] bgmac_poll
    1,11%  ksoftirqd/1      [kernel.kallsyms]    [k] fib_rules_lookup
    1,00%  ksoftirqd/1      [kernel.kallsyms]    [k] sch_direct_xmit
    0,99%  ksoftirqd/1      [kernel.kallsyms]    [k] nf_hook_slow
    0,95%  ksoftirqd/0      [kernel.kallsyms]    [k] __netif_receive_skb_core
    0,91%  ksoftirqd/0      [kernel.kallsyms]    [k] __skb_flow_dissect
    0,80%  ksoftirqd/0      [kernel.kallsyms]    [k] build_skb
    0,76%  swapper          [kernel.kallsyms]    [k] l2c210_inv_range
    0,70%  ksoftirqd/1      [kernel.kallsyms]    [k] __local_bh_enable_ip
    0,64%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_forward
    0,63%  ksoftirqd/0      [kernel.kallsyms]    [k] arm_dma_unmap_page
    0,61%  ksoftirqd/1      [kernel.kallsyms]    [k] netif_skb_features
    0,59%  ksoftirqd/0      [kernel.kallsyms]    [k] dma_cache_maint_page
    0,59%  ksoftirqd/0      [kernel.kallsyms]    [k] bcma_host_soc_read32
    0,56%  ksoftirqd/0      [kernel.kallsyms]    [k] kmem_cache_alloc
    0,55%  ksoftirqd/1      [kernel.kallsyms]    [k] validate_xmit_skb.constprop.53
    0,55%  ksoftirqd/1      [kernel.kallsyms]    [k] page_address
    0,51%  ksoftirqd/0      [kernel.kallsyms]    [k] vlan_do_receive
    0,50%  ksoftirqd/0      [kernel.kallsyms]    [k] mmioset
    0,50%  ksoftirqd/0      [kernel.kallsyms]    [k] page_address

2. Slow (iptable_raw loaded):
Samples: 957K of event 'cycles', Event count (approx.): 331007765065
Overhead  Command          Shared Object        Symbol
    5,69%  ksoftirqd/0      [kernel.kallsyms]    [k] v7_dma_inv_range
    4,68%  ksoftirqd/1      [kernel.kallsyms]    [k] v7_dma_clean_range
    4,46%  swapper          [kernel.kallsyms]    [k] v7_dma_inv_range
    3,73%  ksoftirqd/1      [kernel.kallsyms]    [k] l2c210_clean_range
    3,19%  ksoftirqd/0      [kernel.kallsyms]    [k] l2c210_inv_range
    2,60%  ksoftirqd/1      [kernel.kallsyms]    [k] __netif_receive_skb_core
    2,55%  swapper          [kernel.kallsyms]    [k] l2c210_inv_range
    2,06%  ksoftirqd/1      [kernel.kallsyms]    [k] bgmac_start_xmit
    1,49%  ksoftirqd/1      [kernel.kallsyms]    [k] fib_table_lookup
    1,44%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_rcv_core
    1,34%  ksoftirqd/1      [kernel.kallsyms]    [k] __dev_queue_xmit
    1,13%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_route_input_slow
    0,98%  ksoftirqd/1      [kernel.kallsyms]    [k] fib_rules_lookup
    0,95%  ksoftirqd/1      [kernel.kallsyms]    [k] nf_hook_slow
    0,93%  ksoftirqd/1      [kernel.kallsyms]    [k] sch_direct_xmit
    0,78%  ksoftirqd/0      [kernel.kallsyms]    [k] __siphash_unaligned
    0,77%  ksoftirqd/0      [kernel.kallsyms]    [k] bgmac_poll
    0,76%  swapper          [kernel.kallsyms]    [k] finish_task_switch
    0,72%  swapper          [kernel.kallsyms]    [k] __do_softirq
    0,72%  ksoftirqd/1      [kernel.kallsyms]    [k] __local_bh_enable_ip
    0,71%  swapper          [kernel.kallsyms]    [k] arch_cpu_idle
    0,70%  swapper          [kernel.kallsyms]    [k] bgmac_poll
    0,64%  swapper          [kernel.kallsyms]    [k] __siphash_unaligned
    0,62%  swapper          [kernel.kallsyms]    [k] bcma_host_soc_read32
    0,59%  ksoftirqd/1      [kernel.kallsyms]    [k] ip_forward
    0,58%  ksoftirqd/1      [kernel.kallsyms]    [k] netif_skb_features
    0,55%  ksoftirqd/0      [kernel.kallsyms]    [k] __skb_flow_dissect
    0,54%  ksoftirqd/0      [kernel.kallsyms]    [k] __netif_receive_skb_core
    0,50%  ksoftirqd/1      [kernel.kallsyms]    [k] page_address
    0,50%  ksoftirqd/1      [kernel.kallsyms]    [k] validate_xmit_skb.constprop.53