★ wanayoo — archive 1999 https://github.com/actor-framework/actor-framework/issues/160Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Unit Tests #160

Closed
Neverlord opened this issue Jul 30, 2014 · 13 comments
Closed

Improve Unit Tests #160

Neverlord opened this issue Jul 30, 2014 · 13 comments
Assignees
Labels

Comments

@Neverlord
Copy link
Member

@Neverlord Neverlord commented Jul 30, 2014

The unit tests in CAF could really use some cleanup and a more fine-grained structure. Several tests in fact check multiple things instead of only test one particular feature. @mavam will probably donate his powerful unit testing framework from VAST, which would be a huge step in the right direction.

@mavam mavam self-assigned this Sep 30, 2014
mavam added a commit that referenced this issue Dec 14, 2014
mavam added a commit that referenced this issue Dec 14, 2014
mavam added a commit that referenced this issue Dec 14, 2014
Also minor coding style adjustments.

Relates #160.
mavam added a commit that referenced this issue Dec 14, 2014
mavam added a commit that referenced this issue Dec 16, 2014
mavam added a commit that referenced this issue Dec 16, 2014
It's not possible to use the logger from the unit test framework inside
test cases. To this end, one can use the macros ERROR, INFO, and
VERBOSE.

Relates #160.
mavam added a commit that referenced this issue Dec 16, 2014
mavam added a commit that referenced this issue Dec 16, 2014
This makes it consistent with BOOST_ASSERT. Moreover, we can now use
CAF_REQUIRE in the unit test framework with the same semantics as
BOOST_REQUIRE.

Relates #160.
mavam added a commit that referenced this issue Dec 16, 2014
mavam added a commit that referenced this issue Dec 16, 2014
mavam added a commit that referenced this issue Dec 17, 2014
mavam added a commit that referenced this issue Dec 17, 2014
@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Feb 10, 2015

The testing framework violates the CAF style guide:

  • please rename the "framework" to libcaf_test
  • unit tests include caf/test/unit_test.hpp, not framework/unit.hpp
  • namespace unit { => namespace caf { namespace test {
  • members use m_ prefix instead of _ suffix
  • doxygen blocks use /** ... */ instead of ///

We also need "API compatibility" to the boost testing framework for Boost.Actor (i.e. patching one using find/replace operations only to get the other). I noticed at least one incompatibility: Boost defines BOOST_TEST_MODULE Hello before including the header rather than placing something like CAF_SUITE("Hello") before the first test case (I don't mind if you support both).

Why no CAF_CHECK_EQUAL? That's actually the most used macro and produces better output than CAF_CHECK.

Once it actually uses the CAF coding style (as opposed to the VAST coding style 😉), we can start porting the unit tests, restructure the libraries (src for source files, caf and its subdirectories for headers, and test for the unit tests) and tweak the CMake setup.

@mavam
Copy link
Collaborator

@mavam mavam commented Mar 4, 2015

  • I've addressed all your feedback.
  • The library is also header-only now.
  • Aside from the minor style changes, I've also re-established API compatibility with Boost. The macro CAF_SUITE now needs to be included before caf/test/unit_test.hpp.

Why not CAF_CHECK_EQUAL? Look at the differences:

CAF_CHECK_EQUAL(x, y);
CAF_CHECK_NOT_EQUAL(x, y);
CAF_CHECK_GREATER_THAN(x, y)

as opposed to:

CAF_CHECK(x == y);
CAF_CHECK(x != y);
CAF_CHECK(x > y);

I hope that's convincing enough. 😉

mavam added a commit that referenced this issue Mar 4, 2015
mavam added a commit that referenced this issue Mar 4, 2015
@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Mar 4, 2015

Sorry, your argument is not convincing because you miss the point:

CAF_CHECK(...) can only print that something went wrong. CAF_CHECK_EQUAL(...) can print out what went wrong.

For example, instead of -file-:-line- check failed: a == b, CAF_CHECK_EQUAL can print -file-:-line- error: expected 42, found 23. This extra information is crucial for debugging.

Although header-only has its benefits, I would much rather have it compiled, to be honest. Re-compiling the unit test boilerplate code every time only increases build times unnecessarily. Boost offers both, so maybe you can adopt this behavior if you insist of a header-only use for your purposes. :)

On more thing: In Boost, you have to register your test suites manually to set timeouts: http://www.boost.org/doc/libs/1_35_0/libs/test/example/unit_test_example_03.cpp. Just in case you want to have the timeout configurable in the future. I don't mind to omit this for CAF and simply use a default timeout. Your watchdog setup doesn't work on all platforms, though. For MSVC I had to replace the "detached thread with abort in it" approach with condition variables and mutexes (see https://github.com/actor-framework/actor-framework/blob/develop/unit_testing/test.cpp). Luckily it's a straightforward fix.

@mavam
Copy link
Collaborator

@mavam mavam commented Mar 4, 2015

Have you actually looked at the output of CAF_CHECK? The macro does value interpolation in addition to capturing the argument as text. In your example, CAF_CHECK simply prints something along the lines -file-:-line- error for a == b: 42 == 23.

I'm fine with going back to compiled if we really have to, but do you really think it's going to matter much? It's not a big library, a handful of classes with only 2 compiled ones, and the include footprint would be just reduced by one header file (<thread>) when going compiled. I don't "insist" on either style, but believe that header-only makes the library more accessible to a wider audience.

I'll fix the timeout right now.

@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Mar 4, 2015

Have you actually looked at the output of CAF_CHECK? The macro does value interpolation in addition to capturing the argument as text. In your example, CAF_CHECK simply prints something along the lines -file-:-line- error for a == b: 42 == 23.

Uh... that's quite a lot of under-the-hood-magic. I would prefer a CHECK_EQUAL simply for portability then (and because it is used quite heavily and rewriting each check would be painful). #define CAF_CHECK_EQUAL(a, b) CAF_CHECK( a == b ) should do the trick, right?

Regarding compile time: yes, I do think it matters. Compilation time already is a problem with CAF. Anything that helps reducing it is more than welcome. Maybe you can move the implementation to an _impl header that's then being included in the .cpp as well as in the standalone include?

@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Mar 4, 2015

And sorry for being nitpicky. :)

@mavam
Copy link
Collaborator

@mavam mavam commented Mar 4, 2015

Sure, the CAF_CHECK_EQUAL seems easy to provide. But I don't think it's much manual effort to do the conversion. At the time I branched the code, there were 251 uses of equality checking. Counting the number of commata per line (fgrep CHECK_EQUAL * | sed 's/[^,]//g' | awk '{ print length }' | sort -n | uniq -c), I get:

 209 1
  25 2
  15 3
   2 5

So 209 would be straight text replacement. So the remaining 42 (!) tests may need manual inspection to see whether the == got inserted at the right spot. It looks like when replacing "from the right" is going to be even less, because the RHS is often a fixed value.

I like the idea of the implementation header, will go with that.

mavam added a commit that referenced this issue Mar 4, 2015
mavam added a commit that referenced this issue Mar 4, 2015
mavam added a commit that referenced this issue Mar 4, 2015
@mavam
Copy link
Collaborator

@mavam mavam commented Mar 4, 2015

I've address all of your comments and squashed the commits into a single one.

mavam added a commit that referenced this issue Mar 4, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 4, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 4, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 12, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 12, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 12, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 12, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 12, 2015
Relates #160.
mavam added a commit that referenced this issue Mar 13, 2015
mavam added a commit that referenced this issue Mar 13, 2015
@Hamdor Hamdor assigned Hamdor and mavam and unassigned mavam and Hamdor Mar 16, 2015
@firegurafiku
Copy link
Contributor

@firegurafiku firegurafiku commented Apr 10, 2015

Sorry for a noob question, but do we really need rolling out a new unit testing library? I've checked a ping-pong test and seen nothing but quite ordinary assertions and unreachable code markers. Why not just use something like Catch, for example? It is a simple one-file header-only library which can be linked to the project using (to your choice):

I understand that a lot of work is already done by @mavam, but I don't really want CAF to take the path of Boost or Qt. I hope one day C++ will have a better infrastructure, with some commonly accepted wide-spread libraries instead of lots of reinvented wheels, and I believe that Catch have a great chances to be among them.

@Neverlord

CAF_CHECK(...) can only print that something went wrong. CAF_CHECK_EQUAL(...) can print out what went wrong. For example, instead of -file-:-line- check failed: a == b, CAF_CHECK_EQUAL can print -file-:-line- error: expected 42, found 23. This extra information is crucial for debugging.

Catch can do that with a single assertion macro (to some extent, of course, as it uses some very interesting C++ template magic internally).

@mavam
Copy link
Collaborator

@mavam mavam commented Apr 10, 2015

Thanks for sharing your thoughts on how to improve CAF. Unfortunately there appears to be a disconnect in our goals and what you seem to advocate.

First, we want to minimize the amount of dependencies to external projects. Ideally CAF only requires a compiler, and nothing else. A simple unit test library does not need to have several thousands line of code (Catch has close to 10k and the Boost Test headers alone twice as much). With the current iteration of CAF test, we have a few hundred lines of maintainable, slim C++11 code. When we need to make a change, we just do it quickly. We can choose between a shared/static library as well as a header-only version. This makes it very attractive for downstream projects that depend already on CAF.

Second, we like the Boost community and want both their attention and users. There are multiple efforts underway to make it possible for CAF to replace several subsystems of CAF with Boost. This is why CAF test is API-compatible to Boost Test, albeit to a subset.

As an aside, we don't want tie our hands with specific build frameworks. Biicode appears to get a lot of traction and we're about to finish the integration, but don't want to depend on it. This leaves us with git submoduels and CMake. We use the former only for optional parts of CAF, and the test framework is part of the core. This also makes external projects less attractive.

I've checked a ping-pong test and seen nothing but quite ordinary assertions and unreachable code markers.

The current porting takes place in a different branch. The branch you looked at presumably just contains a brief example illustrating its use.

Catch can do that with a single assertion macro (to some extent, of course, as it uses some very interesting C++ template magic internally).

So does CAF test, by the way, just with much less boilerplate. What are the real advantages of Catch?

@firegurafiku
Copy link
Contributor

@firegurafiku firegurafiku commented Apr 10, 2015

@mavam

Thank you for that detailed answer. I get your opinion and agree with most of your points.
But the only one thing I'm advocating is "do not create one more unit testing framework unless really needed". If your answer is "really", I believe you.

So does CAF test, by the way, just with much less boilerplate. What are the real advantages of Catch?

I've mentioned Catch as one of possible and easy to get running option, but I'm going to feel absolutely comfortable if CAF is going to eventually adopt Boost Test.

As an aside, we don't want tie our hands with specific build frameworks

While that sounds quite reasonable, that would you do if you would replace libcaf_test with Boost Test? You would have to depend on the build system unless you want to duplicate Boost Test source code completely. Problems like that are caused by the absence of the common dependency management infrastructure in C++, and the only one thing seen to take place in the niche is Biicode. I'm not a fan of it too, but there are no alternatives either.

@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Apr 10, 2015

We're not going to switch to Boost.Test for CAF. We are working on a "boostified" version of CAF called Boost.Actor that we propose to the Boost community. For the Boost version, we're going to re-use Boost components like Boost.Test for instance. That's why we need the API compatibility.

The feedback from the Boost community so far was mixed, so we don't know if CAF/Boost.Actor will ever be accepted as a library. In any case, CAF will remain and will be available as standalone version (just like ASIO coexists with Boost.ASIO).

@Neverlord
Copy link
Member Author

@Neverlord Neverlord commented Apr 10, 2015

I close this issue since we are already working on a pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.