Science research is generating extremely large amounts of data--running into petabytes per year--often across distributed locations. At the same time, a large amount of science research is being done across the world with data that has been collected elsewhere. This brings up the issue of effective access to and organization of such data. The increasingly high degree of collaborative research makes this an even more daunting task.

This paper extends the existing framework for sharing and load balancing in this context, with locality-aware allocation of data. This exploits the spatial locality usually observed in data requests.

The HiSbase system that implements this technique uses z-quadtrees to capture spatial locality and to provide efficient access. Z-quadtrees are extensions of quadtrees that use histograms, distributed hash tables, and space-filling curves for linearization. For distributed hash tables, a system called Pastry is used. Histograms are used for capturing locality, and are obtained through training with sample data. The system targets communities with “fairly stable data distributions,” making static training feasible.

The paper covers a wide range of issues relating to the problem of storage, including rate of change of distribution, data skew, implementation of z-quadtrees, node management, data distribution, reliability of nodes, evaluation parameters, and so on. Performance analysis is reported for some large datasets running into millions of records.

The paper reads well and will be of interest to those working with distributed databases, particularly with spatial data, or using data grids.

The lack of focus of the paper--addressing a gamut of issues, instead of focusing on some core elements--makes the paper quite sketchy; a number of questions remain unanswered. At the same time, a lot of familiarity with the work in progress is assumed on the part of the reader, as implied by the lack of adequate background descriptions. Most of the figures are cited without adequate explanation, which makes them difficult to understand.