[Buildroot] [PATCH v3 1/8] support/testing: create default test case for python packages

[Ricardo Martincoski]

Test cases for python packages are very similar among each other: run a
simple script in the target that minimally tests the package.
So create a new helper class named TestPythonPackageBase that holds all
the logic to run a script on the target.

TestPythonPackageBase adds in build time one or more sample scripts to
be run on the target. The test case for the python package must
explicitly list them in the "sample_scripts" property. The test case
then automatically logins to the target, checks the scripts are really
in the rootfs (it calls "md5sum" instead of "ls" or "test" in an attempt
to make the logfile more friendly, since someone analysing a failure can
easily check the expected script was executed) and then calls the python
interpreter passing the sample script as parameter.
An optional property "timeout" exists for the case the sample script
needs more time to run than the default timeout from the test infra
(currently 5 seconds).

A simple test case for a package that only supports Python 2 will look
like this:

|from tests.package.test_python import TestPythonPackageBase
|
|
|class TestPythonPy2<Package>(TestPythonPackageBase):
|    __test__ = True
|    config = TestPythonPackageBase.config + \
|        """
|        BR2_PACKAGE_PYTHON=y
|        BR2_PACKAGE_PYTHON_<PACKAGE>=y
|        """
|    sample_scripts = ["tests/package/sample_python_<package>.py"]
|    timeout = 15

Signed-off-by: Ricardo Martincoski <ricardo.martincoski at gmail.com>
Cc: Arnout Vandecappelle <arnout at mind.be>
Cc: Asaf Kahlon <asafka7 at gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
Cc: Yegor Yefremov <yegorslists at googlemail.com>
---
Changes v2 -> v3:
  - back to v1 but without any tricks and using a separate class;
  - fix typo when using super() TestPythonBase -> TestPythonPackageBase,
    in v2 it ended up calling the same method, but it is wrong;
Changes v1 -> v3:
  - don't use a hack, use __test__ = False from nose2 to declare a class
    that looks like a test case but is not a test case, as consequence
    all classes that inherits from it and are test cases must set
    __test__ = True;
  - create a new class TestPythonPackageBase;
  - add docstring;

Changes v1 -> v2:
  - do not reuse TestPython2 and TestPython3 to for two entirely
    separate things: (Thomas)
    - As a base class for testing individual Python packages;
    - As test cases for the Python interpreter itself;
  - use a better class hierarchy (Thomas). I did this in various patches
    (before and after this one) trying to make the review easier.
  - with the new class hierarchy the trick that allows the defconfig to
    be changed by each test case without explicitly naming the class
    that contains the base defconfig is moved from every test case to
    the new base class.

v1: http://patchwork.ozlabs.org/patch/984425/
---
 .../package/copy-sample-script-to-target.sh   |  7 +++
 support/testing/tests/package/test_python.py  | 56 +++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100755 support/testing/tests/package/copy-sample-script-to-target.sh

diff --git a/support/testing/tests/package/copy-sample-script-to-target.sh b/support/testing/tests/package/copy-sample-script-to-target.sh
new file mode 100755
index 0000000000..6448a80d6d
--- /dev/null
+++ b/support/testing/tests/package/copy-sample-script-to-target.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+set -e
+
+shift
+for file in "$@"; do
+	cp -f "${file}" "${TARGET_DIR}/root/"
+done
diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 26cf49947b..c422bdbf50 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -68,3 +68,59 @@ class TestPython3(TestPythonBase):
         self.math_floor_test()
         self.libc_time_test()
         self.zlib_test()
+
+
+class TestPythonPackageBase(TestPythonBase):
+    """Common class to test a python package.
+
+    Build an image containing the scripts listed in sample_scripts, start the
+    emulator, login to it and for each sample script in the image run the python
+    interpreter passing the name of the script and check the status code is 0.
+
+    Each test case that inherits from this class must have:
+    __test__ = True  - to let nose2 know that it is a test case
+    config           - defconfig fragment with the packages to run the test
+    It also can have:
+    sample_scripts   - list of scripts to add to the image and run on the target
+    timeout          - timeout to the script to run when the default from the
+                       test infra is not enough
+    When custom commands need be issued on the target the method
+    run_sample_scripts can be overridden.
+    """
+
+    __test__ = False
+    config_sample_scripts = \
+        """
+        BR2_ROOTFS_POST_BUILD_SCRIPT="{}"
+        BR2_ROOTFS_POST_SCRIPT_ARGS="{}"
+        """.format(infra.filepath("tests/package/copy-sample-script-to-target.sh"),
+                   "{sample_scripts}")
+    sample_scripts = None
+    timeout = -1
+
+    def __init__(self, names):
+        """Add the scripts to the target in build time."""
+        super(TestPythonPackageBase, self).__init__(names)
+        if self.sample_scripts:
+            scripts = [infra.filepath(s) for s in self.sample_scripts]
+            self.config += self.config_sample_scripts.format(sample_scripts=" ".join(scripts))
+
+    def check_sample_scripts_exist(self):
+        """Check the scripts were really added to the image."""
+        scripts = [os.path.basename(s) for s in self.sample_scripts]
+        cmd = "md5sum " + " ".join(scripts)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+    def run_sample_scripts(self):
+        """Run each script previously added to the image."""
+        for script in self.sample_scripts:
+            cmd = self.interpreter + " " + os.path.basename(script)
+            _, exit_code = self.emulator.run(cmd, timeout=self.timeout)
+            self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        """Test a python package."""
+        self.login()
+        self.check_sample_scripts_exist()
+        self.run_sample_scripts()
-- 
2.17.1