воскресенье, 19 апреля 2015 г.

The mystery of the FitNesse tests order

Everything was started when we added new suite for new test lab. For some reason our Fitness tests began to fail, despite that this suite just link files from another. After some research we understood that was caused by one test: it changed some environment, and that cause other test to fail. In another suites this test was running last,in new suite it runs in the middle.
This strange behavior of FitNesse encouraged me to begin the investigation:

In what order FitNesse runs tests?

So, after some research i understood, that this topic is covered in dark.
Let`s see official documentation:

Remember, that the order tests run is alphabetical.

http://fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.WritingAcceptanceTests.TestSuites.TagsAndFilters

But in service reference we see:

…only tests whose full path names are lexigraphically greater (later in alphabetical order) will be run. This is of questionable use since tests are not guaranteed to be run in any particular order.
http://fitnesse.org/FitNesse.FullReferenceGuide.UserGuide.AdministeringFitNesse.RestfulServices

Hmm. Where is the truth? There is some GitHub issues about this, but they are not clear the situation.

The order in which suites are run is illogical (at least not the same order in which it is presented) https://github.com/unclebob/fitnesse/issues/131#event-122266448

Also i have found great answer from FitNesse author Robert Martin:

How can I get FitNesse to control the order of test execution?


As may folks have said, you can’t. What you can do is !include all
the tests into a single page. Then they will get run as a single test
and in the order of the !includes.

As others have said, this is generally a bad practice, but if you must
do it for some reason, you can.


Fitnesse sorts the child test pages
alphabetically before executing them so they’ll always be executed in
alphabetical order.


Actually FitNesse makes no guarantee about the order at all. If the
order appears alphabetical, that’s a coincidence and not by design.
It will likely change in the next release.
http://fitnesse.996250.n3.nabble.com/Controlling-the-order-of-tests-td9654.html

That’s it, I thought. But why, that is the question.

Why?

And another search gives me this thread:

My experience says, that a suite executes its test in the order as they are stored on the HD - means alphabetically ordered.

I concur. Few people mentioned before the order is alphabetic and it based on the full path to each content.txt
http://fitnesse.996250.n3.nabble.com/Order-of-tests-from-Suite-td11634.html

So everything depends on how Java returns files. If we look at FitNesse source code, we can see, that all magic happens in this lines:

public static File[] getDirectoryListing(File dir) {
    SortedSet<File> dirSet = new TreeSet<File>();
    SortedSet<File> fileSet = new TreeSet<File>();
    File[] files = dir.listFiles();
    if (files == null)
      return new File[0];
    for (int i = 0; i < files.length; i++) {
        ...

According to Java documentation:

public File[] listFiles()

Returns an array of abstract pathnames denoting the files in the directory denoted by this abstract pathname.

There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.
http://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles%28%29

An this is the answer, that i`m looking for.

Conclusion

All tests should be independent. This is very bad practice to rely on tests order, you should avoid it.

Комментариев нет:

Отправить комментарий