[OpenWrt-Devel] Slow DNSMasq with > 100, 000 entries in additional addresses file

[TheWerthFam]

Right now I'd rather not customize the code.  There are two directions 
I'm going to try first.
Give unbound a try to serve DNS, keeping Dnsmasq for DHCP.  If that 
doesn't work try converting the list to a hosts file pointing to a local 
pixelsrv address.  There are some other blog posts that indicate that 
the hosts file can handle a lot more entries.  Like 
https://github.com/pi-hole/pi-hole  Maybe just run pi-hole on openwrt.
Cheers
    Derek

On 12/28/2016 02:21 PM, Dave Taht wrote:
> On Tue, Dec 27, 2016 at 11:03 PM, TheWerthFam <thewerthfam at gmail.com> wrote:
>> Thanks for the feedback, I'll look into NFQUEUE.  I'm forcing the use of my
>> dns by iptables.  I'm also using a transparent squid and e2guardian to
>> filter content.  I like the idea of the dns based blacklist to add some
>> filtering capabilities since I don't want to try and filter https types
>> sites.  I know no solution in perfect.
> I've been thinking about this, and given the large amount of active
> data in a very small memory space have been thinking that another
> approach would be more fruitful. Convert the giant table into a
> "minimally perfect hash", and mmap it into memory read-only, so it can
> be discarded under memory pressure, unlike ipset, squid, or dnsmasq
> based approaches.
>
>
>> Cheers
>>   Derek
>>
>>
>>
>> On 12/27/2016 01:53 PM, philipp_subx at redfish-solutions.com wrote:
>>>> On Dec 26, 2016, at 10:32 AM, TheWerthFam <thewerthfam at gmail.com> wrote:
>>>>
>>>> Using the adblock set of scripts to block malware and porn sites. The
>>>> porn sites list is 800,000 entries, about 10x the number of sites adblock
>>>> normally uses.  With the full list of malware and porn domains loaded,
>>>> dnsmasq takes 115M of memory and normally sits around 50% CPU usage with
>>>> moderate browsing usage.  CPU and RAM usage isn't really a problem other
>>>> than lookups are slow now. Platform is cc 15.05.1 r49389 on banana pi r1.
>>>>
>>>> The adblock script takes the different lists, creates files in
>>>> /tmp/dnsmasq.d/ entries looking like
>>>> local=/domainnottogoto.com/   one entry per line.  The goal is to return
>>>> NXDOMAIN to entries in the lists. Lists are sorted and with unique entries.
>>>>
>>>> I've tried increasing the cachesize to 10,000 but that made no change.
>>>> Tried neg-ttl=3600 with default negative caching enabled with no change.
>>>>
>>>> Are there dnsmasq setting that will improve the performance?  or should
>>>> it be configured differently to achieve this goal?
>>>> Perhaps unbound would be better suited?
>>>>
>>>> Cheers
>>>>      Derek
>>>
>>> Not to rain on your parade, but the obvious defeat of this solution would
>>> be to point to an external website which does DNS lookups for you, and then
>>> edit the URL to have an IP address in place of the host name.
>>>
>>> I would use netfilter’s NFQUEUE and make a user-space decision based on
>>> packet-destination (since it seems you’re filtering outbound traffic
>>> requests).
>>>
>>> After all, it’s not the NAME you don’t want to talk to… it’s the HOST that
>>> bears that NAME.
>>>
>>> -Philip
>>>
>> _______________________________________________
>> openwrt-devel mailing list
>> openwrt-devel at lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>
>
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel