Moderator (MDR): MDR-Greg Voss

Guest Speaker (SPK): SPK-name

MDR-Greg Voss: Hello, everyone. Welcome to the JDC Discussion Forum.This week we'll be talking with Kevin Smith who is a test developer for the Java Compatiblity Kit (JCK). Also joining us is Carla Schroer, Testing Manager for the JCK. Welcome Carla and Kevin. How about starting off with talking about the JCK. What is it, why is it important? What is the good the bad and the ugly in the area of compatibility testing and conformance testing?

SPK-Carla Schroer: The JCK is the tests and test tools we use to ensure that Java implementations work according to the specs. All licensees of Java must pass these tests.This is the how we try to enforce the "write once, run anywhere" story.

SPK-KevinSmith: But, just like the JDK and Java in general, this is a work in progress.

SPK-Carla Schroer: The JCK that will ship to licensees along with the JDK1.1 contains over 8000 tests covering the language spec, VM spec, and API.

SPK-KevinSmith: The good: Lots of tests, especially for the Java Virtual Machine¹ and the Java Language Specification. The bad: Lots more testing to do. The really difficult: testing the AWT across lots of platforms.

MDR-Greg Voss: Since both Kevin and Carla have an interactive style of responding to questions, I'm going to let them feed off of one another's answers here in the early part of the discussion. If anyone has any questions about compatibility and conformance testing, now is a good time to submit them.

SPK-Carla Schroer: Other interesting problems we face in testing are dealing with applications versus applet environments. In an applet environment some operations are restricted for security, yet we strive to have the same API tests cover both situations.

SPK-KevinSmith: Jumping back to GUI testing: Current technology in testing GUI's relies on either hooking into the underlying OS and tracking calls to GUI libraries or screen shots. Calls to a specific OS library are not platform-independent, neither are screen shots platform-independent. Even screen shots for the same platform are not portable. You need a screen shot for each color model for each screen resolution.

SPK-Carla Schroer: So what we do is provide some automated tests and we provide some manual tests that a human being must verify. It isn't perfect and we need more coverage in this area.

MDR-Greg Voss: You have a relatively small team. How are you able to efficiently cover the large number of areas involved in releasing something like the JDK?

SPK-KevinSmith: We work very hard!

SPK-Carla Schroer: We started with 3 people here, and 5 people in Russia one year ago. We built most of the test tools here and really figured out all the requirements for conformance testing. The Russian team started building language tests We've added a few more people here and we have a total of 15 in Russia now (6 in St. Petersberg, 9 in Novosibirsk). Of course with a test suite this size we have to spend quite a bit of time with maintenance, reviewing tests, updating the tools to handle different types of tests, etc.

MDR-Greg Voss: Is it hard working with a team in Russia that is at such a distance? I suppose it must be advantageous in the sense that they are outside the immediate JavaSoft culture and therefore are likely to see and notice different issues. Are there any barriers, or unexpected advantages that come from working with the Russians?

SPK-KevinSmith: One of the benefits of doing test development offsite is that we expose problems in the specifications earlier. The teams in Russia don't have the advantage of being able to chat with the developers, so they rely on written communication. The folks in Novosibirsk have extensive background in testing programming languages, and the folks in St. Petersburg have excellent object oriented design and development background.

SPK-Carla Schroer: There's another real advantage to them being separate, which is that they aren't interrupted by all the "firedrills" in this environment. They have tasks and milestones and they just keep plugging away at it. Also, they are 11 and 14 hours ahead of us, so if we ask a question or need some tests analyzed, they do it in what is "overnight" for us and we have an answer first thing the next day.

mhgreen: Are you doing assertion based testing?

SPK-KevinSmith: We're approaching test development from different points of view for the different parts of Java. Since the Virtual Machine and the Java Language specifications are fairly complete, we have used assertion-based testing for those parts of Java. With the API libraries, it's a little more difficult to just rely on assertion-based testing. We're augmenting assertion testing with a couple more behavior-based test development techniques: equivalence class partitioning, and boundary-value analysis.

MDR-Greg Voss: Interesting to see the strong object-oriented background from the team in St. Petersburg. Any idea why they are so strong in OO? I'm aways curious how certain teams or universities develop to their strengths.

SPK-Carla Schroer: Vladimir Safonov, the manager of the St. Petersberg team is a real expert in OO technology. Most of the people on the St. Petersberg team were students of his at the university where he still teaches a class on OOP.

A couple of other notes on the background of these folks. Sun has been working with this Russian engineering team for over 4 years. Many members of the team were part of the "Russian supercomputer" project before the Soviet government fell. Most of that team was picked up by Sun There are ~150 Russian engineers working on Sun projects in Russia.

MDR-Greg Voss: So how big is it?I've heard a lot about the complexity of our test suite. I'm wondering about the kinds of logistical and tactical problems that managing such a large and complex test suite poses for you.

SPK-KevinSmith: It's soooooo big....

There are about 40,000 files and 15,000 directories. The JCK takes about 60MB of space. However, on some older style file systems, it can expand to take 1.7GB of space. There are about 8500 tests in the test suite.

SPK-Carla Schroer: We've had to build all our own tools. We have to run anywhere where Java runs and we have to be implementation-independent. So we've built everything in Java. We have a test harness application we call "JavaTest" which is how we manage the testing process. Everything from how tests are selected, running tests in different environments, to comparing the results of test runs.

MDR-Greg Voss: You've got some pretty interesting challenges regarding the large number of licensees you have to support. You've got multiple platforms: JavaOS, Windows 95 and NT, Solaris, what else? How do you deal with this complexity?

SPK-Carla Schroer: Well, like I said everything is in Java, which is a good start. The testing process is complex and the tools are somewhat complex. We have to work with the licensees to help them get everything running. We answer quite a bit of support mail from licensees. We also have features that let us do things like run all the tests in a different JVM than where the JavaTest application is running. So the system under test doesn't have to support the whole test environment, just the individual tests. We also have to think about browsers, so we have a version that runs as an applet (with more restricted functionality). In that case, there may not be access to a file system, so we read the class files over using http, and create the results files back on the server. With features like networking we have to support client server testing, so we've had to add some features to manage that piece. It's an ongoing problem, and we continue to refine the tools and the process as we gain more experience.

MDR-Greg Voss: Turbogek has a question about testing for core dumps and testing problems related to recursion and garbage collection. I think there are really three different questions here.

turbogek: Is a core dump (GPF) of the Java VM considered a fail in your tests? I am referring mainly to memory problems created by deep recursion. In general how thorough is the current test suite in regard to testing the speed and reliability of garbage collection?

SPK-KevinSmith: A crash on any platform is usually considered a test failure, but that may depend on the circumstances. JCK is designed to test conformance, not robustness. If some aspect of a Java Virtual Machine works according to the specification, then it should pass the associated tests. With regards to garbage collection, some aspects are not specified, like performance, so we have no conformance tests for that.

On the other hand, we are also a software development company. We're very concerned that our own implementation of Java be as robust as possible, and have the best performance that we can. So we do additional testing of our Java implementations, in addition to just conformance testing.

SPK-Carla Schroer: Compatibility testing isn't really about reliability or stress testing. We strive to make sure that things work according to the spec, but the VM under test can pass each test individually and not have to pass the whole suite at once. So, if a core dump is easy to reproduce and a test causes one, it is a test failure. If the core dump is a result of a stress test, say creating lots of objects, then it wouldn't be a conformance test.

MDR-Greg Voss: What's next in compatibility testing for your team. I have to say, that my hat is off to you for even attempting to tackle the whole area of testing for different browsers and security issues involved? What do you see as your most pressing challenges for the immediate future?

SPK-Carla Schroer: We look at this as an iterative process. With 1.0 there were really minimal tests, which just basically made sure the API was there and did a little language testing. Almost all the Java products on the market today only passed these minimal suites. We shippped our first real JCK , based on 1.0.2 in August and it had mostly language and VM tests. Since then we've focused in the API area, so the JDK 1.1 tests will be much more thorough there. There are still holes of course, we need to focus on the AWT and also Internationalization. It takes us a little while to catch up with the new stuff, especially when we started the project after 1.0 shipped.

SPK-KevinSmith: We would like to find a way to leverage some of the testing technology that already exists to allow us to generate test cases more quickly.

We're working on technology to allow us to test large sections of the AWT library without relying on screen shots or calls to the underlying OS library. We are automating as much of the AWT testing as we possibly can.

SPK-Carla Schroer: It's also really valuable to have real users of the tests who are highly motivated to look at results and report problems to us. We've found a few tests where there was something implementation-independent in the test and we had to rewrite it, or we were making some assumption that wasn't really in the specs. We will continue to maintain what we have, and to write new tests, not just for new features, but to better cover the areas we already have. Sometimes what we find is ambiguity in the specs, so we work with development to get those things addressed as well.

MDR-Greg Voss: It's been fun talking with you, Carla and Kevin. I've got a new appreciation for managing software testing and conformance. "Write once, run anywhere" brings another order of magnitutude of difficulty to an already overwhelming problem. 8500 tests in the suite! Amazing!Thanks for sharing your insights and knowledge.Thanks readers for joining us. See you next week. Last moderator (me) signing off. The forum is now unmoderated.

_______
¹ As used on this web site, the terms "Java virtual machine" or "JVM" mean a virtual machine for the Java platform.