Team Members
Vince Lehman (@vslehman), Davide Pesavento (@Pesa), Eric Newberry (@eric135)
Problem Statement
Currently, the NFD integration tests can take hours to run using either Vagrant or Emulab. It would be helpful to the project if integration tests could be run more efficiently and in a much shorter time on Mini-NDN. In addition, this new testing infrastructure could potentially be integrated with Jenkins to test for breaking changes.
Approach
To effectively use each team member's skills, we worked simultaneously on two tasks:
Vince and Davide worked to get the testing infrastructure and the integration tests working on Mini-NDN.
Eric worked to optimize the Vagrant tests for comparison against the Mini-NDN infrastructure. This was accomplished by eliminating the bottleneck of installing Boost and the NDN libraries and packages four separate times. Now, these steps are executed once on a template VM. This is then cloned for each of the four NDN nodes in the testing environment.
Tasks
- Run sshd on each node
- Be able to ssh from node A to every other node
- Create topology file or hard code topology for integration tests
- Determine which tests pass and which tests fail
- Work towards 100% test success rate
Results
Mini-NDN
Time to create topology and run 9 tests: 2 minutes, 14 seconds
Vagrant
Time to install dependencies, build code, create VMs, and run tests: 84 minutes, 54 seconds
Current Status
Tests that currently pass (9 of 19):
- test_interest_aggregation
- test_unixface
- test_ndnpeekpoke
- test_route_expiration
- test_ndnping
- test_cs_freshness
- test_nrd
- test_fib_matching
- test_ndntraffic
Tests that currently fail:
- test_linkfail ** 2 Failures: multicast and ncc (1 of 3 pass)
- test_hub_discovery ** Passes but many errors are displayed; may be issue with test
- test_interest_loop ** cp: cannot stat ‘/usr/local/etc/ndn/keys’: No such file or directory ** ERROR: error while connecting to the forwarder (No such file or directory)
- test_localhost_scope ** Asks for login info
- test_multicast_strategy **Asks for login info
- test_multicast ** nfd-status command does not run
- test_tcp_udp_tunnel ** A can’t create Face to D; no route
- test_localhop ** Error while connecting to the forwarder
- test_nfdc ** Error while connecting to the forwarder
- test_remote_register ** Asks for login info
Vagrant
The optimizations to the Vagrant environment have reduced the run time significantly. Instead of taking three or four hours, the integration tests now take just under an hour and a half to run.
TODOs
- Get all tests working!
Future
- Trigger tests from Jenkins
- Add command-line option to run specific integration test