The Data Intensive  Cyberinfrastructure Foundation

2010 iRODS User Group Meeting

Wed, 19 May 2010 13:10:08 -0700

iRODS User Group Meeting 2010 Website
Proceedings of iRODS User Group meeting 2010 (PDF)
Proceedings of iRODS User Group meeting 2010 on Amazon (ISBN 1452813426)

The second annual iRODS user group meeting was held from March 24-26, 2010 in Chapel Hill, North Carolina. The first meeting was held in Nice, France in February 2009. The meeting attracted a wide range of attendees from experienced software developers who are contributing code to the open source iRODS software, to users with installations of varying sizes, to potential users wanting to learn more about iRODS. Users from academia, international and business organizations, and federal and state agencies were represented at the meeting.

With its advanced design and highly configurable and extensible architecture, iRODS is gaining increasing attention and users. The meeting attendance exceeded capacity with attendees from across the United States and as far away as France, Australia, and Taiwan, and participation from Japan by teleconference.

The call for participation encouraged submissions on a wide range of topics, including user applications of iRODS, interoperatibility and clients for accessing the iRODS data system, and integration with other data and repository systems. Contributions include papers, posters, and slides (available on the meeting website at https://www.irods.org/index.php/iRODS_User_Meetings).

The meeting began with sessions on an introduction to release 2.3 of iRODS, descriptions of how to create new micro-services and policies, a review of the unix shell command capabilities (icommands), and descriptions of the interactions (queries) on the iCAT metadata catalog.

A majority of the meeting was devoted to sessions for papers and presentations on user applications. The papers demonstrated the wide range of ways that communities have applied the iRODS framework. More than a dozen presentations and posters described use of iRODS in national data grids (e.g. Australian Research Collaboration Service), large-scale scientific research projects (e.g. KEK high energy physics data grid, French national computing center CC-IN2P3), institutional repositories (e.g. Carolina Digital Repository), and preservation environments (e.g. Taiwan National Archives).

Additional sessions were devoted to the development of new iRODS clients and integration of new capabilities into the iRODS framework. In particular, the JARGON Java I/O library was presented along with plans for integration with web services, development of a dropbox interface (iDrop), and support for digital library interfaces such as Islandora.

The final day of the meeting was dedicated to seeking community input into prioritizing new feature development, and identifying new types of interoperability mechanisms that are needed. A list was compiled of the current clients used to interact with an iRODS data grid. More than 35 existing clients were identified, ranging from Grid clients (GridFTP, JSAGA), file system interfaces (FUSE, PetaFS, webDAV), GUI interfaces (iExplore), digital library interfaces (Fedora, DSpace, Drupal, Islandora), workflow systems (Kepler, Taverna) and specialized interfaces such as URSpace for synchronizing local resources with iRODS. Interactions between collaborators are being supported through the Data Intensive Cyberinfrastructure Foundation, which supports the iRODS open source community.

Bridging Federal Data Repositories for Science and Society

Sun, 18 Apr 2010 17:49:31 -0700

As technology advances, growing networks of sensors and sophisticated computer models are generating torrents of data. This wealth of information data makes possible vital insights into urgent science and engineering challenges from climate change to predicting floods, developing drought-resistant plant species, and more.

But until now, the diverse data collections have largely remained isolated islands or “silos.” This makes it difficult for scientists from different fields to share this rich data across disciplinary lines – an indispensable step for weaving together the strands of their distinct disciplines into an integrated fabric of knowledge.

In a recent demonstration of “connecting the dots” across diverse repositories, nine collaborating Federal and state projects working with the Data Intensive Cyber Environments Center (DICE Center) at the University of North Carolina (UNC) at Chapel Hill have been “federated” or connected together. This interconnection, which allows discovering and moving data between them, uses an advanced NSF supported “data grid” system known as iRODS, the Integrated Rule-Oriented Data System.

Bridging these repositories will enable key national data-driven science, new collaborative research, and management of the full data life cycle. For example, time dependent climate data collected in the new NSF Ocean Observatories Initiative (OOI) can be stored and preserved in the National Oceanic and Atmospheric Administration (NOAA) National Climatic Data Center (NCDC) in Asheville, NC, the researchers explained. From the NCDC repository, this climate data can be accessed for use in hydrology research at the Renaissance Computing Institute (RENCI) on severe flooding events, combined with hydrology data from the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) preserved at UCSD’s San Diego Supercomputer Center.

The repositories have also been federated with supercomputing centers including RENCI and the Texas Advanced Computing Center (TACC). This lets scientists compare new climate data with older archived records to detect major changes, and also use climate observations in supercomputer model simulations to drive major improvements in predictions of future events like regional climate change, severe flooding events, and more.

Connecting climate with biology, NCDC climate data can in turn be accessed in this iRODS-based federated system by the iPlant Collaborative in partnership with TACC, helping iPlant study the genetic basis of designing drought-resistant plants so that agriculture can adapt to changing climate. The data federation capability also opens novel dimensions of cross-discipline collaboration, such as connecting the Odum Institute for Research in Social Science at UNC. Odum can now interoperate with, for example, CUAHSI hydrology data, linking social science data on how people are impacted by and respond to hydrological events such as floods.

Other iRODS-based repositories federated in this effort include the RENCI visualization data grid at UNC, which works closely with the DICE Center; and CIBER-U engineering education collections. For long term preservation, Federal agency records can flow to the National Archives and Records Administration (NARA) Transcontinental Persistent Archives Prototype (TPAP).

In the big picture, the researchers added, scientific research is becoming increasingly collaborative, and the ability to federate and interconnect digital data collections is now a fundamental requirement for advancing science and engineering in the US and around the globe.

by Paul Tooby, DICE Communications

Related links:
Data Intensive Cyber Environments Center (DICE Center) http://dice.unc.edu
Integrated Rule-Oriented Data System (iRODS) https://www.irods.org
Renaissance Computing Institute (RENCI) http://www.renci.org
The Texas Advanced Computing Center (TACC) http://www.tacc.utexas.edu
Ocean Observatories Initiative (OOI) Cyberinfrastructure http://ci.oceanobservatories.org
NOAA National Climatic Data Center (NCDC) http://www.ncdc.noaa.gov
Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) http://www.cuahsi.org/
iPlant Collaborative http://www.iplantcollaborative.org
Odum Institute for Research in Social Science http://152.2.32.107/odum/jsp/home.jsp
Cyber-Infrastructure-Based Engineering Repositories for Undergraduates (CIBER-U) http://gicl.cs.drexel.edu/wiki/CIBER-U
Data Intensive Cyberinfrastructure Foundation http://diceresearch.org
University of California, San Diego http://www.ucsd.edu/
National Archives and Records Administration (NARA) http://www.archives.gov/ncast
National Science Foundation (NSF) http://www.nsf.gov/

iRODS Version 2.3 with Extensible iCAT, Quotas, and More

Tue, 16 Mar 2010 13:24:54 -0700

The developers of iRODS, the Integrated Rule-Oriented Data System, have announced the release of version 2.3 of the open source software. With its “intelligent cloud” architecture, iRODS is a leading technology for creating, sharing, and preserving sharable collections of digital data, no matter where the data are distributed -- across the hall or around the world.

Development of the open source iRODS system is led by the Data Intensive Cyber Environments Center (DICE Center) at the University of North Carolina at Chapel Hill, with major development at the Institute for Neural Computation (INC) at UC San Diego.

Version 2.3 can be downloaded, along with user information and the release notes from the iRODS website http://irods.org.

The iRODS project has pioneered many functions that make possible “virtual” shared collections, hiding the diversity and complexity of the storage layer from users, as in cloud storage. Taking this one step further, the advanced iRODS middleware system also provides powerful rules and policy-based automation that make it feasible to create, share, manage, and preserve today’s massive stores of digital information which can range from personal data collections on a laptop computer to petabytes of data in hundreds of millions of files distributed around the globe.

New features in iRODS 2.3 include an extensible iCAT system to make it easier for sites to add tables into the iCAT database and make use of those tables via standard irods-client capabilities.

Another new capability is an optional quota feature that maintains and enforces quotas on how much data users and groups can store into iRODS as a whole or on specific resources. Quotas are enforced through a rule and micro-service.

Version 2.3 adds support for both GSI and Kerberos at the same time, and users can now select Kerberos, GSI, or iRODS-password authentication. Thanks to Roger Downing of the UK e-Science project for providing these extensions.

The performance of the iCAT has been improved for large numbers of collections through a new DBMS index. Support has also been added for Groupadmin users who can now add and remove other users from groups to which they belong.

Progress statistics have also been added to iRODS. For example, for iput and iget the progress of transferring a large file will be reported, and using this feature the Windows iRODS explorer will now report the progress of operations.

Support for Collection Soft Links has been added as well, allowing a soft link to be made from one collection to another, and supporting links across zones and multiple links in a path.

Jargon 2.3.0 has also been released as part of the general iRODS 2.3 release, and is available from the iRODS SVN.

iRODS is funded by the National Archives and Records Administration and the National Science Foundation.

Related links
• Integrated Rule-Oriented Data System (iRODS) https://www.irods.org
• Release Notes 2.3 https://www.irods.org/index.php/Release_Notes_2.3
• Data Intensive Cyber Environments Center (DICE Center) at UNC http://dice.unc.edu
• UCSD Institute for Neural Computation (INC) http://inc.ucsd.edu
• Data Intensive Cyberinfrastructure Foundation http://diceresearch.org
• UNC School of Information and Library Science (SILS) http://sils.unc.edu
• Renaissance Computing Institute (RENCI) http://www.renci.org

NARA’s Robert Chadduck Receives 2010 Federal 100 Award

Tue, 23 Feb 2010 14:53:31 -0800

Robert Chadduck, Principal Technologist for Advanced Research at the National Archives and Records Administration (NARA) Center for Advanced Systems and Technologies (NCAST) , has received a 2010 Federal 100 Award from Federal Computer Week. This award recognizes 100 individuals in government and industry who have made major contributions to the federal information technology community and furthered research in the IT world in the past year. Awardees are nominated by Federal Computer Week readers and selected by a panel of IT experts.

As the Principal Technologist for Advanced Research at NCAST, Mr. Chadduck's responsibilities include:
* Identifying leading edge technological challenges faced by the National Archives;
* Identifying research projects that could address these challenges;
* Forming alliances with other agencies in jointly sponsoring research projects;
* Ensuring that researchers understand the unique aspects of NARA's needs;
* Overseeing the research and communicating results; and
* Advising agency management on technical issues.

NARA is a long time partner of the DICE group in developing innovative technologies for preservation and sharing of large collections of digital data, and sponsors core development of iRODS, the Integrated Rule-Oriented Data System.

For more information on the work of the National Archives Center for Advanced Systems and Technologies, visit: http://www.archives.gov/ncast.

The Data Intensive Cyberinfrastructure Foundation

2010 iRODS User Group Meeting

Bridging Federal Data Repositories for Science and Society

iRODS Version 2.3 with Extensible iCAT, Quotas, and More

NARA’s Robert Chadduck Receives 2010 Federal 100 Award

The Data Intensive  Cyberinfrastructure Foundation