Contributions to the development of a national grid intrastructure for E-science.

Abstract

e-Science is a terminology denoting modern scientific experiments and studies being carried out with the support of large-scale Grid infrastructures. The essence of a Grid is to enable resource sharing, including compute resources and storage resources, to as many authorised people as possible. This thesis has made several contributions towards building a national grid infrastructure, by designing and implementing new approaches to simplify the use of the Grid so as to enlarge the user base of grid infrastructure and e-Science. These new systems have been deployed as part of the Australian national grid infrastructure, however the approaches used aim to provide a generic solution so that other grid operators and users can also benefit from it. Our exploration of existing data grid systems has shown that these are not easy for researchers to use, since they require users to have certain IT knowledge, and rarely have a user-friendly interface. My approach is to develop a web portal based upon a widely-used data grid system, iRODS, that is able to make use of geographically distributed storage resources. This new interface not only supports the WebDAV standard, enabling easy drag-and-drop file access, but also provides a web interface, allowing users to share and manage data with a web browser. Data transfer is a challenge when dealing with large volumes of data and long distances, which leads to problems with stability, reliability and performance. Existing data repositories and data transfer services deliver necessary functionality, but only support a limited number of protocols, which can cause problems with interoperability. Rather than developing a new data transfer service or modifying current services to support new protocols, my approach focuses on equipping an arbitrary data source with astandard GridFTP interface, so that it can interact with most of the existing data transfer services and grid services. This thesis gave a detailed description of my architecture and evaluation, and demonstrated that this approach adds virtually no overhead to the data source but gives it more flexibility in data transfer. Compute job submission usually requires users to have a significant level of understanding of the Grid, such as its structure and the usage of its client tools, especially when users are exposed to a complex grid infrastructure with multiple resources. The client tools and interfaces are not easy to use or to develop custom applications. My approach addresses this problem by providing a web portal with a RESTful interface to simplify job submission to multiple grid resources. The RESTful interface also makes it possible for users and application developers to submit massive jobs in a simple way. The portal has a template system to enable quick and easy development of customiszd interfaces to applications running on grid compute resources. The portal therefore provides a generic solution to users across various research domains.