[Buildroot] [PATCH] support/scripts/check-host-libs: add new check on host binaries/libs

[yann.morin at orange.com]

Thomas, All,

On 2022-09-20 10:35 +0200, Thomas Petazzoni spake thusly:
> On Tue, 20 Sep 2022 09:46:24 +0200
> <yann.morin at orange.com> wrote:
[--SNIP--]
> > Finally, this script takes more than a minute to run on our build, about
> > a 10% increase from ~12min. This is not nice at all.
> >     $ find host/*bin host/lib* -type f |(keep just libs + execs) |wc -l
> >     339
> > That last command, though, is what takes time: there is a huge ton of go
> > junk installed in HOST_DIR, and this takes ages to filter-out.
> Wow, 1 extra minute, this seems a lot. However, I don't understand what
> you mean by "That last command". What exactly takes time? The fact that
> we run "file" on zillion of files to filter out files that are not
> executable/shared libraries? Or the readelf on the filtered files?

Sorry, I was not explicit enough. Yes, the filtering only, i.e. only
running 'file' on all those 16k+ files takes more than a minute:

    $ date +%s; find host/*bin host/lib* -type f |while read f; do
        mime=$(file -b --mime-type ${f})
        if test "${mime}" != "application/x-sharedlib" -a
                "${mime}" != "application/x-executable" ; then
            continue
        fi
        printf '%s\n' "${f}"
    done |wc -l; date +%s

    1663659605
    339
    1663659681

I.e. 76 seconds just to identify the files to actually look at.

> > So, although I understand the rationale, this should probably be opt-in.
> > 
> > Or it should only be ran when doing an SDK?
> 
> I am fine with this being opt-in, but I would not tie it to the SDK,
> but rather to CI in the autobuilders. Indeed, while locally for your
> own projects where you might control the build environment (using
> containers, as you mentioned), in the general situation, Buildroot
> tries to not use system libraries other than the C library. So having
> this in the autobuilders would not impose the extra time on users, but
> would allow us to detect a number of undetected spurious host
> dependencies.
> 
> So, it would be a Config.in option, disabled by default. Autobuilders
> would enable it, and users who are interested by the extra checks could
> also enable it.
> 
> Thoughts?

Yes, being opt-in is probably the best solution, in the "Build options"
-> "Advanced" submenu.

Or we can see at optimising it. Overall, I am not a fan of "if it's too
slow, don't do it", but I prefer "if it's too slow, make it faster".

So, we can start by spawning less processes, then use sed to filter out
the result:

    find ${HOST_DIR}/*bin ${HOST_DIR}/lib* -type f -print 0 \
    |xargs -0 -r file --mime-type \
    |sed -r -e '/^(.+): application/(x-executable|x-sharedlib)$/!d; s//\1/' \
    |while read f; do
        readelf blabla...
    done

I'll try to find some free CPU cycles to look further into that soon...

Regards,
Yann E. MORIN.

-- 
                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |
'--------------------------------------:______/_____:------------------'


_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.