Objective of OpenWRT/x86?

[Elliott Mitchell]

On Thu, May 04, 2023 at 03:50:05AM +0100, Daniel Golle wrote:
> On Wed, May 03, 2023 at 06:36:10PM -0700, Elliott Mitchell wrote:
> > On Tue, May 02, 2023 at 02:45:43AM +0200, Alberto Bursi wrote:
> > > 
> > > 
> > > On 26/04/23 22:17, Elliott Mitchell wrote:
> > > > Well, was a specific objective ever chosen for the x86 version of
> > > > OpenWRT?
> > > > 
> > > > I can state my goal/hope for OpenWRT/x86.  The *WRT Linux distributions
> > > > were so named for originally targeting the LinkSys WRT54G.  This was a
> > > > small AP, so one might expect builds to be for small APs.  By today's
> > > > standards 128MB RAM and 16MB of storage is relatively small.
> > >  >
> > > 
> > > Afaik the x86_64 images approach has always been "include what you can 
> > > to make it run on everything without need to customize the image".
> > > Storage is integrated in kernel because otherwise it won't see the 
> > > rootfs and fail to boot, everything else is in modules (possibly) but 
> > > modules for most ethernet controllers are always included in default build.
> > > 
> > > Then there is the virtualized drivers that were just added to the kernel 
> > > and not properly modulized, maybe it was easier like that, don't know.
> > 
> > These two are the same issue.  Not including the drivers for a hypervisor
> > could make the image not boot on that hypervisor.  Though most
> > hypervisors will include some fallback device emulation which provides
> > inferior performance.
> > 
> > > Because on x86_64 storage and RAM is measured in Gigabytes so saving a 
> > > few hundred kilobytes by removing the "all hardware just works with a 
> > > single image" feature is NOT a worthy tradeoff.
> > 
> > Partially true, partially untrue.  Saving a few hundred kilobytes isn't
> > worthwhile.  Shrinking the kernel by 35% and userspace by 20% though IS
> > worthwhile.
> > 
> > Notably I'm making use of Xen.  The Xen devices don't try to look like
> > normal hardware.  As a result the Xen block devices don't depend on the
> > SCSI subsystem.  Then you completely disable all emulated graphics
> > devices (simulated serial consoles work fine) and things have shrunk
> > drastically.
> 
> Kernel size on x64/64 is 5780.8 KB. Even if you really manage to shrink
> the kernel by 35%, that would mean 2023 KB. That's 1.5% of the total
> memory available on a 128 MB system...

Something is very wrong here.

Question is whether it is deliberate versus accidental.

One possibility is you last looked at an OpenWRT x86 5.10 kernel and
figured the number would be roughly in line with 5.15.  In this case
there was a rather drastic increase in size between 5.10 and 5.15 for
x86.

Another possibility which comes to mind is that is pretty close to the
x86 5.15 compressed image size.  I would expect you to be aware modern
data compression algorithms are fairly effective and can often provide
substantial reductions.  I suppose you could be unaware of this, but
given how well known this is I would be left suspecting deliberate
distortion.

The last generic OpenWRT x86 5.15 image I built less than a week ago
produced a roughly 28MB vmlinux.bin file.  Some of that disappears since
initialization portions will be freed after boot, but this is fairly
representative of runtime use (additional will be used for what is found
during boot).

My most recent OpenWRT based image which was moderately adapted to a
specific target (running in a Xen VM) was 18MB.  *That* is substantial
enough to go after.


> > > > Examples of things which don't seem to fit well are CONFIG_AGP and
> > > > CONFIG_DRM.  I would expect those for a desktop Linux distribution due
> > > > to GUI.  For OpenWRT which tends towards networking/embedded those seem
> > > > out of place.  CONFIG_FB is similar, though some devices do have actual
> > > > limited frame-buffers.
> > > 
> > > I don't know exactly how much impact this has but on many x86 systems 
> > > it's very nice to have the video ports work so you can connect a screen 
> > > and interact with console. On some older boards or laptops it might be 
> > > over AGP bus so dropping this would drop their screen support.
> > > 
> > > For reference, other network appliance projects like pfSense and 
> > > OpnSense also support basic text console on video ports so that's 
> > > something many "expect".
> > > 
> > > Now, it's probably all ancient stuff that not a lot of people are using. 
> > > So yeah ok remove it from x86_64 but leave it on the x86 and the legacy 
> > > images because that's where people with ancient hardware might need it.
> > 
> > AGP support only effects graphics performance.  Without the support AGP
> > devices lose some performance, but continue to work (since they continue
> > to act as PCI).
> 
> That's simply not true. AGP support is required to allocate video memory
> from shared system memory, which on some systems without any dedicated
> video memory boilds down to the ability to setup a frame buffer which
> matches the native resolution of the screen (ie. DKMS on i915, to name
> the most popular example). I've mentioned that already in a previous
> reply to your suggestion to remove AGP support (which, in terms of
> software, means support for shared video memory).

I had been left with the (very possibly incorrect) understanding on such
systems the framebuffer area was generally reserved by the BIOS.  Yet
again though, back to my earlier point is OpenWRT/x86 aiming to be a
desktop Linux distribution?

If the answer to that is "no" then being able to setup a framebuffer
isn't an issue.


> > Many desktops can use a serial port as console (universal on servers).
> > Hypervisors generally provide some flavor of serial port-like device
> > (often they misbehave by doing rather more than 115.2kbps).
> 
> ... and many cheap thin clients or systems like the Intel NUC which
> are popular as high-performance network appliances come with on-board
> graphics relying in shared video memory, and many people will want to
> use HDMI+USB as a console due to the lack of a serial port (I know
> it's available on some pinheader inside the case, but not even a
> standard connector...)

Yet again, I'm asking the question:  What is THE goal of OpenWRT/86?

Is OpenWRT/x86 aiming to be a desktop Linux distribution?  Is OpenWRT/x86
a developer test simulator for "real" OpenWRT targets?  Is OpenWRT/x86 a
networking device Linux distribution?



On Thu, May 04, 2023 at 05:54:08AM +0200, Stefan Lippers-Hollmann wrote:
> 
> On 2023-05-03, Elliott Mitchell wrote:
> > On Tue, May 02, 2023 at 02:45:43AM +0200, Alberto Bursi wrote:
> > > On 26/04/23 22:17, Elliott Mitchell wrote:  
> [...]
> > > Then there is the virtualized drivers that were just added to the kernel 
> > > and not properly modulized, maybe it was easier like that, don't know.  
> > 
> > These two are the same issue.  Not including the drivers for a hypervisor
> > could make the image not boot on that hypervisor.  Though most
> > hypervisors will include some fallback device emulation which provides
> > inferior performance.
> > 
> > > Because on x86_64 storage and RAM is measured in Gigabytes so saving a 
> > > few hundred kilobytes by removing the "all hardware just works with a 
> > > single image" feature is NOT a worthy tradeoff.  
> > 
> > Partially true, partially untrue.  Saving a few hundred kilobytes isn't
> > worthwhile.  Shrinking the kernel by 35% and userspace by 20% though IS
> > worthwhile.
> > 
> > Notably I'm making use of Xen.  The Xen devices don't try to look like
> > normal hardware.  As a result the Xen block devices don't depend on the
> > SCSI subsystem.  Then you completely disable all emulated graphics
> > devices (simulated serial consoles work fine) and things have shrunk
> > drastically.
> [...]
> 
> And combining these two statements, you're in a catch-22 dilemma...
> If you optimize the kernel configuration for Xen-only, it will no longer
> boot on qemu-kvm, hyper-v, virtualbox, vmware. Once you resurrect support
> for all of those, your high-density virtualization goals go out of the
> window - and the gap to support for real hardware becomes negligible, 
> especially once you add in support for IOMMU/ DMAR based PCI pass-through
> support (which was already mentioned to pass through wireless cards).

Well, it turns out we've already got that catch-22 dilemna since an
Atheros79 image won't boot on a Raspberry PI 4B.

I don't consider this a problem since an image shrunk for Xen isn't meant
to boot on Qemu.  While an image targetting for Qemu isn't likely to work
on Xen.  Nor will a Hyper-V image work on Qemu or Xen.

For ARM or MIPS, each of these would be a separate subtarget, yet
everything has been jumbled together on x86.  I'm surprised Geode has its
own subtarget.


All the NICs are kernel modules and that is appropriate for this use
case.


> > I'm concerned at the large number of things enabled in the kernel
> > configurations.  Traditionally OpenWRT has used f2fs, yet x86 also has
> > ext4.  With the percentage of systems using SSDs, would seem advantageous
> > to stick with that flash-friendly choice.
> [...]
> 
> f2fs && ext4 is a bad example here, on the one hand for the reasons Daniel
> Golle has already raised (the selection between f2fs and ext4 being based
> on the overlay size at firstboot time), on the other hand because we are 
> only talking about a few KB in this case (and f2fs is a horrible 
> filesystem, huge overhead (way more than half of those 100 MB given as 
> deciding factor), bad fsck options, bad (online-)resize behaviour, so 
> killing f2fs with fire would be the more sensible approach - not forgetting
> that dropping ext4 support would kill the ext4 based images, which are very
> popular among x86 and ARM SBC (RPi, sunxi, rockchip) users).

I checked by building kernels.  EXT4 and F2FS are roughly 2MB each.  I'm
up for removing F2FS instead, but having both is wasteful.


> > Many desktops can use a serial port as console (universal on servers).
> > Hypervisors generally provide some flavor of serial port-like device
> > (often they misbehave by doing rather more than 115.2kbps).
> 
> Serial console ports (in the sense of rs-232c) on consumer desktops became 
> extinct around sandy-bridge times, consumer notebooks haven't had them for 
> over two decades by now. Yes, in some cases you will find a non-standard
> onboard header for them, but even that isn't a given. Yes, I'm aware that
> business computers are more likely to sport serial consoles (even today)
> and that dedicated x86_64 routing platforms and rack servers are very 
> likely to have them, but those aren't the only (very common-) x86 system.
> None of your consumer (desktop-) mainboards/ system will, none of your
> notebooks will either - and it takes two to tango (admitted, the client
> side may go with USB2serial adapters, but the host side won't be able to).

I guess this varies with what you classify as "consumer".  The desktop
motherboard makers of note (ASRock, ASUS, Gigabyte, MSI, others) all
include a standard by fiat serial header.  Given how reliable the
inclusion is I'm pretty sure it is on all the chipsets, though
manufacturers may omit appropriate headers for some markets.


> for x86_64/generic. If you really wanted to provide high-density demands,
> you'd end up with at least five (kvm, Xen, hyper-v, virtualbox, vmware, 
> parallels, maybe more) new images next to x86_64/generic, which I don't 
> think anyone really wants or needs (those who do, will want even more
> fine-tuned images, and x32 may be part of those considerations). And no,
> I'm not advocating for this, at all.

Yup, that list is most of them.  Though some of those could end up stuck
on combined targets for a while.

KVM/Qemu, Hyper-V and Xen are likely the bigger gains.  My impression is
the Virtio stuff is fairly encompassing and thus a drag on others.  While
Hyper-V is pretty complex and also a major drag on others.  Meanwhile Xen
is very small and thus gets dragged down by others.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445