Computing Reviews
Today's Issue Hot Topics Search Browse Recommended My Account Log In
Review Help
Search
A survey on artifacts from CoNEXT, ICN, IMC, and SIGCOMM conferences in 2017
Flittner M., Mahfoudi M., Saucez D., Wählisch M., Iannone L., Bajpai V., Afanasyev A. ACM SIGCOMM Computer Communication Review48 (1):75-80,2018.Type:Article
Date Reviewed: Nov 18 2019

This paper does precisely what the title says. It is an attempt to survey the current state of reproducibility in computer networking research. This is one of the more challenging areas of computing for reproducibility, in that more specific hardware, particularly at the Information-Centric Networking (ICN) conference, tends to be involved.

The methodology was to retrospectively question all of the authors of the 137 papers from the named conferences, and then to look at the answers (49 papers) and directly at the artifacts. Hence, this paper is much more than a survey. The completion rate (36 percent) and the likelihood of differential completion mean that the statistics have to be taken with a pinch of salt. It is a pity that the authors did no analysis of the nonresponders. Did they actually have published artifacts, and/or should they?

One key finding is “less than 20 [percent] of researchers store artifacts on their personal or project website; instead they use popular public code platforms such as GitHub.” While ideally the figure would be zero percent, the general feeling is that “less than 20 [percent]” is better than many fields would show.

Another key finding is that “a large number of papers in our survey used already existing public datasets and testbeds or infrastructures.” The authors rightly commend IMC for giving a community contribution award “that recognizes a paper with an outstanding contribution to the community in the form of a novel dataset, source code distribution, open platform, or other noteworthy service to the community.”

All is not rosy:

Only [a] minority of papers ([that is], three publications) provide scripts to produce figures or compute numerical data, which is presented in the papers. In the majority of the papers, artifacts do not cover 100 [percent] of the results in the paper.

Nevertheless, the picture painted is that this community (or at least the fraction that responded) is healthily addressing the reproducibility challenge (see also [1]).

Reviewer:  J. H. Davenport Review #: CR146776 (2002-0031)
1) Saucez, D.; Iannone, L. Thoughts and recommendations from the ACM SIGCOMM 2017 Reproducibility Workshop. ACM SIGCOMM Computer Communication Review 48, (2018), 70–74.
Bookmark and Share
  Featured Reviewer  
 
Data Communications (C.2.0 ... )
 
 
General (A.0 )
 
 
Reference (A.2 )
 
Would you recommend this review?
yes
no
Other reviews under "Data Communications": Date
Communications formulas & algorithms for systems analysis and design
Rorabaugh C., McGraw-Hill, Inc., New York, NY, 1990. Type: Book (9780070536449)
Feb 1 1992
Telecommunications for management
Meadow C. (ed), Tedesco A., McGraw-Hill, Inc., New York, NY, 1984. Type: Book (9780070411982)
Jan 1 1985
After the breakup
Crandall R., The Brookings Institution, Washington, DC, 1991. Type: Book (9780815716068)
Jun 1 1992
more...

E-Mail This Printer-Friendly
Send Your Comments
Contact Us
Reproduction in whole or in part without permission is prohibited.   Copyright 1999-2024 ThinkLoud®
Terms of Use
| Privacy Policy