Test improvement – More and better tests for LibreOffice

One of the areas that can help LibreOffice, but may not directly be visible to the users even though it has a big impact is the quality assurance, is automated testing.  Here, I discuss some areas and notes around improving tests for LibreOffice. First, I start with regressions and bug fixes without a test.

Missing Unit Tests

This is the description from the GSoC ideas page:

While there are some automated tests for LibreOffice, there are not nearly enough. Adding more and better tests helps developers who work on the code to be more productive by allowing them to find regressions as early as possible.

To elaborate more, I should say there are many regressions and bugs in general that are fixed, but lack testing. It is important to have tests for those bug fixes, to avoid such problems in the future. You can see a list of those fixed issues that lack tests here:

• The Document Foundation Wiki – CppUnitTest
• The Document Foundation Wiki – Missing Unit Tests

If you want to add some new tests for the bug fixes, first you should read the bug report very carefully to understand what is it about, and try to test the fix yourself. You can either use git revert command to revert the fix, and see the problem in action, or try changing back the fix in the code, if it is not too big.

That is important, because you have to see that the test fails without the fix in place, but succeeds when it is applied. This is an essential step when writing the test.

To know more about unit tests, you may look into these Wiki pages:

• The Document Foundation Wiki – UITests

Porting Existing Test to C++ or Python

Some tests are written in the past, but now have issues because of the way they are written. In this case, porting them can provide improvement.

Tests can be written in multiple languages. At least, C++, Python, Java and BASIC are currently in use for different tests across the LibreOffice code base. Again in the GSoC ideas page, you can read:

There is some support in LibreOffice for automated tests, both at the level of unit tests, and at the level of system tests that drive a full LibreOffice instance. Currently tests can be written in C++, Java, or Python. Various new automated tests should be developed to improve the test coverage.

Tests written exclusively for Java (e.g. the JUnit framework) should be ported to C++ so that they can execute much more rapidly. Similarly, tests that do remote control of an existing LibreOffice instance, should be re-factored to run inside that instance to make debugging much easier.

Almost all JUnitTests and UITests, and also some smoke tests, run as an outside process. To verify, the trick is to remove the soffice(.exe) binary, or to remove its execute permission (on Linux). In this way, out-of-process tests should fail, as they need to run the soffice binary. After a successful build, you can see if this works. For example, try this:

make -d JunitTest_framework_complex

As an example, we have this issue around complex tests that should be ported to Python:

• tdf#97362 – Convert Java complex tests to Python

You may find many examples of the previously ported test in the issue page. Keep in mind that usually old Java tests should be removed in the same patch that provides the CppUnitTest port.

Also, you may find examples of UITests ported to C++ from Python. For example, see this patch:

• tdf#150457 Port UITest to CppunitTest

It ports a previously implemented UITest from Python to C++. The result is a faster test, which is more robust.

Focusing on a Specific Area

If you want to add or port some tests, please focus on a specific area of the code. The LibreOffice code base is huge, and consists of more than 200 modules. It is not easy, and in fact not suggested, to work across different applications and modules. It is much better that you pick a specific application, and even inside that application, like Writer, Calc, Impress, etc., focus on a specific module. In this way, it would be much easier to understand the ideas and the way tests are implemented.

How Hard is it to Write / Port a Test?

One important question is around the complexity of writing or porting tests. The answer is not straightforward, as it depends on the area that the test is written. But one can take a look at what others did in the past. By looking into Gerrit or git log, you can find what other people, including GSoC applicants, were able to achieve during their work.

After writing a few tests, or porting a few tests, it may become easier for you to write more tests. Another thing is that you don’t have to write tests from scratch. There are many tests available in the LibreOffice code base, and you can create your new test based on existing tests and customize it to match the purpose of the issue you are working on.