Data Intensive Cyber Environments Research

Announcing iRODS Version 2.0: Uniting Collaborations

Tue, 9 Dec 2008 17:27:22 -0800

Federation Capability Supports Nuanced Data Sharing for Collaborative Research

The Data Intensive Cyber Environments (DICE) group has announced the release of version 2.0 of iRODS, the Integrated Rule-Oriented Data System. The new version of the award-winning software adds a number of important features, including federation of independent iRODS installations which lets them “talk” to each other, supporting large-scale collaboration by giving users seamless access to data distributed across different iRODS systems.

Core development of the open source iRODS data system is led by the Advanced Center for Data Intensive Cyber Environments at the Institute for Neural Computation at the University of California, San Diego and the National Center for Data Intensive Cyber Environments at the School of Information and Library Science at the University of North Carolina at Chapel Hill. Download of version 2.0, user information, and release notes are freely available as open source software from the iRODS wiki.

“A major new feature in iRODS 2.0 is the ability to federate two or more independent iRODS data grids,” said Reagan Moore, director of the Data-Intensive Computing Environments group and professor in SILS at UNC. “Federation lets communities maintain independent iRODS installations, while choosing to share some or all of their data under explicit management policies.” iRODS does this by mapping the policies to computer-actionable rules that control all remote operations as well as data exchange between separate iRODS systems or Zones. Additional federation iRODS Rules are applied on top of the local Rules at each iRODS data grid.

There will be an iRODS workshop February 2-5, 2009 that will bring together both researchers new to iRODS with others using iRODS in a range of applications. Online registration is free and open through the January 10, 2009 deadline.

iRODS moves beyond the single-site repository model, which is based on the traditional hard copy paradigm, to implement a new paradigm that harnesses the full power of cyberinfrastructure and the virtual world to free digital data collections from the constraints of space -- whether physical, administrative, or disciplinary – and time, through long-term preservation. This approach gives users an adaptable and extensible system with the integrated capabilities required for the full range of digital data management applications, from highly customizable sharing in data grids, to publication of data in digital libraries, sensor stream aggregation for real-time data systems, and long term preservation of digital data for use in standard reference collections.

New features in iRODS version 2.0 include:
• iRODS Zone Federation. Each separate iRODS installation or “iRODS Zone” – which consists of one or more iRODS Servers, a single associated iRODS Metadata Catalog, and multiple Clients – can share data and metadata.
• Master/Slave iCAT. An iRods Zone can be configured to run with a single Master iCAT metadata catalog plus optional Slave iCATs synchronized with the Master catalog. This can reduce latency, speeding up metadata queries across wide area networks.
• iRODS Explorer for Windows client provides a rich Graphical User Interface and fast navigation and operations to manage data.
• SRB to iRODS Migration Tool. This preliminary version of a migration tool helps convert an SRB instance to an iRODS one, letting the iRODS system access the data formerly under SRB management, without the need to move the physical files.
• A new bundling feature gathers large numbers of small files into structured files such as tar files for efficient uploading, downloading, and archiving.

“The iRODS 2.0 release contains many new features and improvements, large and small, based on user requests and our years of experience with iRODS and the SRB Storage Resource Broker,” said Senior Software Developer and Designer Wayne Schroeder. “In the aggregate these make iRODS a highly capable system that equips users to solve a wide variety of data management problems by making use of various subsets of the features.”

iRODS supports seamless growth from small installations to the largest scales. At UCSD alone iRODS and the previous Storage Resource Broker (SRB) technology are already managing 1.2 petabytes of data and two hundred million files for 5,000 users, and growing.

“We also understand that performance is a very important part of iRODS usability, especially at the larger scales, and in addition to the new federation capability this release also contains important performance enhancements,” said iRODS Software Architect Mike Wan. “We’ve added an efficient mechanism for transferring large files, a bundling mechanism for transferring a large number of small files, and a caching enhancement.”

Other features of interest include the addition of a number of new micro-services; improvements in iRODS use of Grid Security Infrastructure (GSI), allowing regular iRODS users to authenticate with GSI; performance improvements in the iRODS FUSE user level file system capability; support for Rule-oriented Data Access to Oracle databases; a new data transfer mode for larger files, RBUDP (Reliable Blast UDP), in addition to the existing sequential (single TCP stream) and parallel (multi TCP streams) modes; and improvements to the iCAT iRODS Metadata Catalog, including rollback after errors to allow execution of subsequent SQL functions in PostgreSQL. iRODS 2.0 also includes improvements in testing and installation scripts.

iRODS version 2.0 is supported on Linux, Solaris, Macintosh, and AIX platforms. The iRODS commands are also supported on the Windows operating system, and there is a Windows GUI client. The iRODS Metadata Catalog (iCAT) will run on both the open source PostgreSQL database (which can be installed as part of the iRODS install package) and Oracle. And iRODS is quick and easy to install -- just answer a few questions and the install package automatically sets up the system for you.

iRODS was first released in late 2006. Version 1.0 of the software was released under a BSD open source license in January 2008. As a second generation data grid development effort, iRODS leverages more than 10 years of user-driven experience with the Storage Resource Broker (SRB). With a grant-funded core developer team, the iRODS system is growing rapidly as collaborating projects contribute code to the open source software.

The iRODS team is working with partners in a number of projects to apply the technology, including the Transcontinental Persistent Archives Prototype (TPAP) for the National Archives and Records Administration (NARA), the Ocean Observatories Initiative (OOI), the NSF Temporal Dynamics of Learning Center (TDLC), the NHPRC-supported Distributed Custodial Archival Preservation Environments (DCAPE) project, the French National Library, and many others.

Collaborators in the iRODS project include the French Institut National de Physique Nucléaire et de Physique des Particules (CC-IN2P3), the Sustaining Heritage Access through Multivalent ArchiviNg (SHAMAN) project, the UK e-Science Data Management Group at Rutherford Appleton Laboratory, and the High Energy Accelerator Research Organization, KEK, in Japan.

In addition to Moore and Rajasekar, the DICE group includes software architect Mike Wan and senior developer Wayne Schroeder, along with Sheau-Yen Chen, Lucas Gilbert, Chien-Yi Hou, Antoine de Torcy, Paul Tooby, and Bing Zhu. SILS professor Richard Marciano leads the DICE Sustainable Archives and Library Technologies (SALT) lab at UNC.

iRODS is funded by the National Archives and Records Administration and the National Science Foundation under the following grants:

- NSF SDCI 0721400, "SDCI Data Improvement: Data Grids for Community Driven Applications" (2007-2010)
- NSF/NARA OCI-0848296, "NARA Transcontinental Persistent Archive Prototype" (2009-2012)

Related links
Integrated Rule-Oriented Data System (iRODS) https://www.irods.org
iRODS Workshop http://indico.in2p3.fr/conferenceDisplay.py?confId=1234
Data-Intensive Cyber Environments (DICE) group http://diceresearch.org
UNC School of Information and Library Science (SILS) http://sils.unc.edu
UCSD Institute for Neural Computation (INC) http://inc2.ucsd.edu

Sustainable Preservation That Works

Fri, 7 Nov 2008 11:29:06 -0800

Distributed Custodial Archival Preservation Environments

In the Information Age society produces a flood of electronic records. State governments, universities, and all sectors of society now rely heavily on records that spend most if not all of their life in electronic form.

But the freedoms that give digital materials their dazzling power also make them ephemeral and challenging to preserve. Despite the fact that electronic records have been around for decades archivists still face a serious gap between the need to archive this growing stream of records in digital form, and the real-world capabilities to make this a reality.

Now, a broad new collaboration, the Distributed Custodial Archival Preservation Environments project, or DCAPE, is taking an important step toward meeting the needs of archival repositories for trusted archival preservation services by giving archivists the tools they need to ensure that software-dependent electronic records created today will be usable with tomorrow’s technology.

The innovative two-and-a-half year project, supported by the National Historical Publications and Records Commission (NHPRC), includes state archives in California, Kansas, Michigan, Kentucky, North Carolina, and New York; university archives at Tufts University and Carleton University (Canada); a cultural institution, the Getty Research Institute; and cyberinfrastructure partners at West Virginia University, the Renaissance Computing Institute (RENCI), and the UNC School of Information and Library Science (SILS). DCAPE is also building on the experience of partners who have worked together in previous projects such as the Persistent Archives Testbed (PAT).

“Many archival repositories are under-funded and struggling to fulfill their responsibilities to preserve and provide access to electronic records,” said project leader Richard Marciano, director of the Sustainable Archives and Library Technologies (SALT) Lab in the Data Intensive Cyber Environments group, Professor in the School of Information and Library Science at the University of North Carolina at Chapel Hill, and Chief Scientist, Persistent Archives and Digital Preservation at RENCI. “So our goal in the DCAPE collaboration is to demonstrate a working cost-effective preservation system and show archivists that this capability may be closer than they think.”

Once practical preservation systems are in place it is clear that the cost to store a unit of information will continue to plummet for electronic records, compared to the ever-rising costs of maintaining hard copy records. As part of this transition, the DCAPE collaboration is developing a sustainable business model based on open source preservation infrastructure.

“We’ll keep the costs far less than most people would expect in several ways,” said Marciano. “We’ll minimize labor costs by automating administration of the preservation environment with iRODS, the Integrated Rule-Oriented Data System.” The iRODS system, developed by the Data Intensive Cyber Environments (DICE) group at the School of Information and Library Science (SILS) and RENCI at UNC Chapel Hill and UC San Diego, has a powerful Rule Engine that lets archivists automate labor-intensive procedures. As freely available open source software, the iRODS system is advancing rapidly through a growing community of users who collaborate on its development.

“Together our DCAPE project includes 33 participants across 12 institutions,” said Marciano. “We can take advantage of this partnership to reduce the risk of data loss and save resources through distributed archives, for example, by using multiple sites in the DCAPE preservation environment to back up each other’s data.”

DCAPE will consist of a distributed and easily expanded preservation environment that enables archivists to manage their records, located at both the DICE facility at UNC and partner institutions in a unified “virtual environment.” Ensuring redundancy, replicas of records may be stored at multiple locations. And by collaborating with each other, DCAPE lets resource-strapped archives reduce the need to invest in so much in-house infrastructure and expertise.

The project will also minimize storage costs through using low-cost commodity storage and a small subset of the massive storage systems that support research at RENCI, the Renaissance Computing Institute at the University of North Carolina.

The DCAPE project will address the risk of technological obsolescence by taking advantage of iRODS’ “infrastructure independence,” the ability to seamlessly migrate electronic records onto newer, more cost-effective infrastructure that continually appears. While archivists can start small with iRODS, in the future they can seamlessly expand their archives to terabytes or petabytes of data.

iRODS, supported by NARA and the NSF, incorporates more than a decade of award-winning research, and offers archivists state-of-the-art automated rules-based capabilities to appraise, describe, accession, replicate, manage, and provide access to large, complex electronic records collections long-term.

For more information on DCAPE see http://www.dcape.org.

Related Links
Data Intensive Cyber Environments (DICE) group http://diceresearch.org
Integrated Rule-Oriented Data System (iRODS) https://www.irods.org/
UNC School of Information and Library Science (SILS) http://sils.unc.edu
Sustainable Archives and Library Technologies (SALT) http://salt.diceresearch.org

Learning About IRODS: Workshop Registration Opens

Mon, 13 Oct 2008 13:21:44 -0700

The Data Intensive Cyber Environments (DICE) group and CC-IN2P3, the Computing Center of the National Institute of Nuclear and Particle Physics (Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules) in France, have announced a workshop on iRODS, the Integrated Rule-Oriented Data System.

The four-day workshop will be held February 2-5, 2009 at CC-IN2P3 in Lyons, France. Online registration is free and open through the January 10, 2009 deadline. This is a pre-registration form, and due to limited space, accepted registrations will be confirmed by e-mail. Early registration is suggested.

iRODS is advanced open source data grid technology for managing, sharing, publishing, and preserving digital data collections, ranging from personal collections to the largest scales -- petabytes of data with hundreds of millions of files.

Growing out of a decade of user-driven experience with the Storage Resource Broker (SRB) software, core iRODS development is being led by the Data Intensive Cyber Environments (DICE) group at the School of Information and Library Science (SILS) and Renaissance Computing Institute (RENCI) at the University of North Carolina at Chapel Hill (UNC) and the Institute for Neural Computation (INC) at the University of California, San Diego.

The open source iRODS data grid system is growing in popularity, and a number of independent projects such as the CC-IN2P3 in France and others in the U.S. and around the globe are using iRODS and developing code with the goal of extending the open source iRODS system.

Chaired by research engineer Jean-Yves Nief of CC-IN2P3 and Reagan Moore, DICE group director, the upcoming workshop will bring together researchers new to iRODS who want to learn more about iRODS and data grid technology, as well as others already using iRODS in a range different communities.

The workshop will open with presentations from users on applications of iRODS technology. Those interested in presenting should contact Jean-Yves Nief. For examples of how people are applying the versatile iRODS technology see the DICE website.

The workshop will also include a tutorial on an Introduction to the iRODS Data Grid System, and a tutorial on Rules and Micro-services. These two key features of iRODS let users translate their high-level management policies into powerful Rules that automate data management operations. The rules control the execution of the lower-level Micro-services at the distributed servers where data are stored.

On the fourth day there will be parallel sessions to help communities with common interests explore and build collaborations to apply and develop iRODS for their specific needs. Planned community sessions tentatively include Astrophysics, Medical Records, Arts and Humanities, and Preservation Environments.

The core iRODS development by the DICE group in the U.S. is supported by the National Science Foundation (NSF) and the National Archives and Records Administration (NARA). Additional development of the open source iRODS data grid system is being supported by CC-IN2P3 and a number of other independent projects worldwide.

For more information, see the workshop website.

About IN2P3
The Computing Center of the National Institute of Nuclear and Particle Physics (CC-IN2P3) is part of IN2P3 in Lyon, France, funded by the French government. Established in 1971, IN2P3’s mission is to promote and conduct research in nuclear and high energy physics. It coordinates programs in these areas on behalf of the CNRS and universities, in partnership with CEA. IN2P3 also brings its expertise to other scientific fields, helps solve scientific and technical problems for society, disseminates expertise and technological advances to industry, and participates with universities to educate the next generation of researchers.

About iRODS and DICE
The Data Intensive Cyber Environments (DICE) group leads core development of the open source iRODS Integrated Rule-Oriented Data System. With more than a decade of award-winning research in advanced technologies for managing, sharing, publishing, and preserving digital data, the group is based at the School of Information and Library Science and the Renaissance Computing Institute (RENCI) at the University of North Carolina at Chapel Hill, and the Institute for Neural Computation at the University of California, San Diego. Development of the core iRODS data grid system is funded by the National Science Foundation and the National Archives and Records Administration, with a growing open source iRODS community participating in development worldwide. For more information see http://diceresearch.org.

Related Links
• iRODS Workshop http://indico.in2p3.fr/conferenceDisplay.py?confId=1234
• CC-IN2P3 Computing Center of the National Institute of Nuclear and Particle Physics http://cc.in2p3.fr
• DICE Data Intensive Cyber Environments group http://diceresearch.org
• iRODS Integrated Rule-Oriented Data System wiki https://www.irods.org
• UNC School of Information and Library Science (SILS) http://sils.unc.edu
• Renaissance Computing Institute (RENCI) http://www.renci.org
• UCSD Institute for Neural Computation (INC) http://inc2.ucsd.edu
• National Science Foundation (NSF) http://www.nsf.gov/
• National Archives and Records Administration (NARA) http://www.archives.gov

DICE Group Receives Society of American Archivists Award

Thu, 7 Aug 2008 15:52:46 -0700

The Society of American Archivists has announced that the Data-Intensive Cyber Environments (DICE) Group has been selected to receive the 2008 J. Franklin Jameson Archival Advocacy Award. The award will be presented August 29th at a ceremony during the association’s 72nd annual meeting in San Francisco.

The award honors an individual, institution, or organization that promotes greater public awareness, appreciation, or support of archives. The DICE Group was selected for its long-time support of and involvement in the archives profession’s work to address the major challenges of managing, preserving, and providing access to electronic records.

The DICE group is at the School of Information and Library Science at the University of North Carolina at Chapel Hill and the Institute for Neural Computation at the University of California, San Diego.
The DICE group has played a leading role in efforts to develop and implement international standards related to electronic records; been partners for more than a decade in digital preservation efforts funded by the National Science Foundation, the National Archives and Records Administration, and the Institute of Museum and Library Services; provided technical expertise to National Historical Publications and Records Commission grant projects; contributed to the archival literature; and provided invaluable support in the many and varied archival research efforts in developing electronic records archives. In January, 2008, the DICE Group released version 1.0 of iRODS, the Integrated Rule-Oriented Data System, an advanced open-source system for managing, sharing, and preserving electronic records.

A hallmark of successful efforts in digital preservation is a collaborative, cross-disciplinary approach, and in making its selection the award committee noted, “Members of the DICE Group have a genuine interest in and understanding of the archives profession, its principles and practices, its unique challenges, and have become strong advocates in its favor.”

Established in 1989, the award is named for the noted American historian J. Franklin Jameson, who was a long-time advocate for the establishment of a National Archives in the United States. Previous recipients include: the Chicago Tribune; Senator Harry Reid (D-Nev); and George F. Farr, Jr., of the National Endowment for the Humanities.

Founded in 1936, the Society of American Archivists is North America’s oldest and largest national archival professional association. SAA’s mission is to serve the educational and informational needs of more than 5,100 individual and institutional members and to provide leadership to ensure the identification, preservation, and use of records of historical value.

For more information about the Society of American Archivists visit www.archivists.org.
For more information about the DICE group see http://diceresearch.org.