[Buildroot] [PATCH 1/1] support/scripts/pkg-stats: iterate over CVEs in streaming

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot at gmail.com> writes:

 > Hi Titouan,
 > El jue., 20 feb. 2020 a las 19:27, Titouan Christophe
 > (<titouan.christophe at railnova.eu>) escribió:
 >> 
 >> The NVD files that are used to build the list of CVEs affecting
 >> Buildroot packages are quite large (a few hundreds MB of json),
 >> and cause the pkg-stats scripts to have a huge memory footprint
 >> (a few GB with Python 2.7).
 >> 
 >> However, because we only need to iterate on CVE items one by one,
 >> we can process them in streaming (ie decoding one CVE at a time
 >> from the JSON representation). Because the json module from the
 >> python standard library does not support such a mode of operation,
 >> we switch to the third-party package ijson, which is compatible
 >> with both Python 2 and Python3.
 >> 
 >> To run the script with these modifications, one should install
 >> the ijson python package. This can be done with pip:
 >> `pip install ijson`. On Debian based distributions, this can
 >> also be done with the apt package manager:
 >> `apt install python-ijson`.
 >> 
 >> Signed-off-by: Titouan Christophe <titouan.christophe at railnova.eu>

 > This is _way_ better. In my test run observing top output, resident
 > memory stayed around 50 MB.

Nice!

 > Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>
 > Tested-by: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>

Committed, thanks.

-- 
Bye, Peter Korsgaard