Currently, the NFD integration tests can take hours to run using either Vagrant or Emulab. It would be helpful to the project if integration tests could be run more efficiently and in a much shorter time on Mini-NDN. In addition, this new testing infrastructure could potentially be integrated with Jenkins to test for breaking changes.
To effectively use each team member's skills, we worked simultaneously on two tasks:
Vince and Davide worked to get the testing infrastructure and the integration tests working on Mini-NDN.
Eric worked to optimize the Vagrant tests for comparison against the Mini-NDN infrastructure. This was accomplished by eliminating the bottleneck of installing Boost and the NDN libraries and packages four separate times. Now, these steps are executed once on a template VM. This is then cloned for each of the four NDN nodes in the testing environment.
- Run sshd on each node
- Be able to ssh from node A to every other node
- Create topology file or hard code topology for integration tests
- Determine which tests pass and which tests fail
- Work towards 100% test success rate
Time to create topology and run 9 tests: 2 minutes, 14 seconds
Time to install dependencies, build code, create VMs, and run tests: 84 minutes, 54 seconds
Tests that currently pass (9 of 19):
Tests that currently fail:
- test_linkfail ** 2 Failures: multicast and ncc (1 of 3 pass)
- test_hub_discovery ** Passes but many errors are displayed; may be issue with test
- test_interest_loop ** cp: cannot stat ‘/usr/local/etc/ndn/keys’: No such file or directory ** ERROR: error while connecting to the forwarder (No such file or directory)
- test_localhost_scope ** Asks for login info
- test_multicast_strategy **Asks for login info
- test_multicast ** nfd-status command does not run
- test_tcp_udp_tunnel ** A can’t create Face to D; no route
- test_localhop ** Error while connecting to the forwarder
- test_nfdc ** Error while connecting to the forwarder
- test_remote_register ** Asks for login info
The optimizations to the Vagrant environment have reduced the run time significantly. Instead of taking three or four hours, the integration tests now take just under an hour and a half to run.
- Get all tests working!
- Trigger tests from Jenkins
- Add command-line option to run specific integration test