The Data Intensive  Cyberinfrastructure Foundation

iRODS 2011 User Group Meeting Announced

Wed, 15 Dec 2010 13:48:04 -0800

The Data Intensive Cyber Environments Center (DICE Center) at the University of North Carolina at Chapel Hill has announced the third annual User Group Meeting for iRODS, the Integrated Rule-Oriented Data System, on “Sustainable Policy-Based Data Management, Sharing, and Preservation.” The meeting, which will be held February 17 - 18, 2011, is being cosponsored by the Renaissance Computing Institute (RENCI) and will be held at the RENCI Europa Center in Chapel Hill.

The user meeting is an opportunity for the growing iRODS community to participate in sessions on applications of iRODS, discuss sustainability initiatives, and learn about new planned technology development. The sessions are all focused on helping users implement and extend the new paradigm of sustainable policy-based data management for the sharing and preservation of today’s diverse and rapidly growing digital data collections.

Meeting organizers invite papers and posters on applications of iRODS, client interfaces, integration examples, and more, with submissions due January 17, 2011. More information on the meeting, the call for papers and posters, and registration is available on the iRODS wiki at https://www.irods.org/index.php/iRODS_User_Group_Meeting_2011.

iRODS is advanced open source data grid technology for creating shareable “virtual” digital data collections, which can range from personal collections to the largest scales -- petabytes of data with hundreds of millions of files -- and span sites distributed across the hall or around the globe.

The iRODS User Meeting will bring together users new to iRODS who want to learn how to get the most out of the advanced technology with others already using iRODS and developers in the open source project.

Because iRODS provides key data management features not found in other open source systems, there is rapid growth in use of the software. Responding to this demand, a new collaboration between iRODS@RENCI and the DICE team is working to expand production support for new iRODS user communities and sustainable development support for features requested by current and future iRODS users. The user meeting will provide an opportunity to interact with both DICE and RENCI staff.

The meeting will feature sessions on:
- New features in iRODS and new versions of the Rule Engine
- Applications of iRODS in data grids, digital libraries, archives
- iRODS interfaces including Jargon and user-supported clients
- Sustainability plans for iRODS open source software
- User feature request prioritization, and more.

About iRODS
Based on more than a decade of user-driven experience with real-world challenges in a range of applications, core iRODS development is led by the Data Intensive Cyber Environments Center at UNC, the UCSD Institute for Neural Computation (INC), and RENCI with major collaboration and partners in the US and internationally.

Since its introduction in 2008 the iRODS data grid system has steadily grown in users, with more downloads with each release. Making use of the significant set of iRODS generic capabilities, projects across the U.S. and international collaborations are applying the versatile iRODS system in a wide range of fields, from sharing data in scientific research collaborations to managing real-time data streams, publishing collections in digital libraries, and preserving electronic records over the long term in archives.

Core development of iRODS is led by the DICE team at UNC, UCSD and RENCI. As the iRODS open source community expands, more projects and developers are contributing code with the goal of extending the open source iRODS system for their applications and the benefit of all. Development partners include CC-IN2P3, the Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules; SHAMAN, the EU Sustaining Heritage Access through Multivalent ArchiviNg project and the University of Liverpool; ARCS, the Australian Research Collaboration Service; UK e-Science; CeRch, the Centre for e-Research at King's College, London; the KEK High Energy Accelerator Research Organization in Japan, and others.

Core iRODS development has been supported by the National Science Foundation (NSF), the National Archives and Records Administration (NARA), and most recently RENCI.
Related Links:
iRODS User Group Meeting 2011 https://www.irods.org/index.php/iRODS_User_Group_Meeting_2011
iRODS Integrated Rule-Oriented Data System wiki https://www.irods.org
Data Intensive Cyber Environments Center (DICE Center) http://dice.unc.edu
Renaissance Computing Institute (RENCI) http://www.renci.org
Data Intensive Cyberinfrastructure Foundation http://diceresearch.org
iRODS Introductory Information http://diceresearch.org/DICE_Site/Introduction.html

CI-BER project plans to store billions of electronic records

Mon, 15 Nov 2010 12:34:29 -0800

The result of a recent three-way partnership between the University of North Carolina at Chapel Hill, National Archives and Records Administration (NARA) and the National Science Foundation (NSF), the Cyberinfrastructure for Billions of Electronic Records project (CI-BER) held its official kickoff meeting on Thursday, Oct. 28, 2010 in Chapel Hill.

One of the first federally-funded projects of its kind to bring the benefits of social media and public reports of its progress via a blogging format, CI-BER will create master copies of NARA's Center for Advanced Systems and Technologies' testbed collections of electronic records and store them at the University of North Carolina at Chapel Hill.

CI-BER involves research partnerships between the School of Information and Library Science (SILS), the Renaissance Computing Institute (RENCI) and the Data Intensive Computing Environments (DICE) Center, as well as with staff at the University of California, San Diego (UCSD). SILS professor Dr. Richard Marciano, leads the UNC portion of the project.

"CI-BER brings together computer scientists, engineers and archival scientists and is meant to provide insights into the management of very large scientific data collections," writes Marciano on the CI-BER project's blog. "We expect the findings to inform other NSF Office of Cyberinfrasdtructure (OCI) funded projects as well."

A second-generation research collaboration made possible by $395,000 in seed funding, CI-BER extends earlier work done under a project called Transcontinental Persistent Archives Prototype (TPAP). Its goal is to further the understanding of digital infrastructure and provide insights into the management of scientific data.

In addition to Marciano, those working on the project include: Stan Ahalt, director of RENCI; Arcot Rajasekar, SILS professor and DICE Center director of research and technology, and chief scientist; Leesa Brieger, head of iRODS@RENCI; Jefferson Heard, senior research software developer, RENCI; Chien-Yi Hou, SILS research associate; and Sheau-Yen Chen, data grid system administrator, Institute for Neural Computation, UCSD.

"CI-BER is a learning opportunity as well as a valuable research project," said RENCI Director Stan Ahalt. "What we learn from this project will be applicable to managing large scientific data sets in general and to developing the kind of generalized cyberinfrastructure that will help our nation manage and archive the tremendous amounts of data being created every day."

To read more about CI-BER, you can follow the CI-BER project's blog at: http://ci-ber.blogspot.com

iRODS Version 2.4 Released

Mon, 16 Aug 2010 14:16:00 -0700

Robust, High Performance Sharing and Preservation of Distributed Data at Scale

The developers of iRODS, the Integrated Rule-Oriented Data System, have announced the release of version 2.4 of the open source software. With its innovative architecture, iRODS is a leading technology for policy-based creation, sharing, and preservation of massive digital collections, no matter where data are distributed -- across the hall, around the world, or in the cloud.

Version 2.4 can be downloaded, with user information and the release notes, from the iRODS website http://irods.diceresearch.org.

iRODS architecture is well proven in large-scale production use in applications around the world, from massive research data collections in physics, astronomy, oceanography, and humanities, to use in digital preservation, digital libraries, and more, and version 2.4 continues the emphasis on robust, high performance at the largest scales.

iRODS technology lets users easily create unified “virtual” shared collections of distributed data, hiding the diversity and complexity of distributed storage from users. Taking this to the next level, further innovations in iRODS middleware let users automate collection management based on local and community Rules and Policies that support creation, management, sharing, and preservation of today’s mushrooming stores of digital information, from personal data collections on a laptop computer to multi-petabyte collections with hundreds of millions of files distributed around the globe.

A key to iRODS steady growth and award-winning innovation has been the developers’ history of listening to users and incorporating new features requested by researchers, archivists, and others facing real-world data management problems. Version 2.4 continues this approach, adding a number of capabilities requested at the second annual iRODS User Group Meeting 2010 and elsewhere.

New features in 2.4 include improved performance for bulk upload and registration which lets users upload 50 files with a single command for faster ingestion of large numbers of small files, a feature requested by users in the archives and astronomy communities, among others. A new Connection Control and Monitoring System (CCMS) provides a set of features that offer additional control of client connections to iRODS servers for managing Denial of Service attacks. This includes multi-threading of iRODS Servers, and the ability of the iRODS administrator to monitor and control the number of connections to all iRODS Servers both locally as well as in a Federation of multiple iRODS Zones, or catalog instances.

Reflecting its development history in collaboration with the supercomputing community as well as the archives and digital library communities, iRODS has long been known for its ability to scale to petabyte collections with hundreds of millions of files. In “extreme scaling” testing in collaboration with users ingesting some 50 million files with 250 million metadata attributes, the developers made a number of improvements in version 2.4 including improved memory use and speeding metadata queries, including additional indexes in the iCAT Metadata Catalog, all of which further extend iRODS performance at the largest scales.

The core open source iRODS software is developed by the Data Intensive Cyber Environments (DICE) research group in the DICE Center at the University of North Carolina at Chapel Hill and the Institute for Neural Computation (INC) at UC San Diego, in close collaboration with the Renaissance Computing Institute (RENCI) and the UNC School of Information and Library Science, and collaborations with the international iRODS open source community.

The iRODS open source community continues to grow, with more partners participating, leveraging the development investment and speeding progress of the software. The 2.4 release includes features resulting from collaborative use, testing, and/or code contributions from the Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules (CC-IN2P3); the French National Library (BnF); the NASA Center for Computational Sciences (NCCS); the Australian Research Collaboration Service (ARCS); the National Center for Atmospheric Research (NCAR), and others.

The DICE research effort has been working with a number of partners and projects to apply the technology, including the National Archives and Records Administration (NARA); the Institute of Museum and Library Services (IMLS) and partners including DuraSpace in the "Policy-Driven Repository Interoperability (PoDRI)" project; the NSF Temporal Dynamics of Learning Center (TDLC); the NSF TeraGrid; the Ocean Observatories Initiative (OOI), and many others.

Additional international collaborators in the iRODS project include the EU Sustaining Heritage Access through Multivalent ArchiviNg (SHAMAN) project and the University of Liverpool; the UK e-Science Data Management Group at Rutherford Appleton Laboratory; the Centre for e-Research at King's College, London; and the High Energy Accelerator Research Organization, KEK, in Japan.

The nonprofit Data Intensive Cyberinfrastructure Foundation serves as the home of the growing iRODS open source community.

Development of the core iRODS system has been funded by the National Science Foundation and the National Archives and Records Administration.

Related links
• Integrated Rule-Oriented Data System (iRODS) https://www.irods.diceresearch.org
• Release Notes 2.4 https://www.irods.org/index.php/Release_Notes_2.4
• Data Intensive Cyber Environments Center (DICE Center) at UNC http://dice.unc.edu
• Renaissance Computing Institute (RENCI) http://www.renci.org
• UCSD Institute for Neural Computation (INC) http://inc.ucsd.edu
• UNC School of Information and Library Science (SILS) http://sils.unc.edu
• Data Intensive Cyberinfrastructure Foundation http://diceresearch.org

2010 iRODS User Group Meeting

Wed, 19 May 2010 13:10:08 -0700

iRODS User Group Meeting 2010 Website
Proceedings of iRODS User Group meeting 2010 (PDF)
Proceedings of iRODS User Group meeting 2010 on Amazon (ISBN 1452813426)

The second annual iRODS user group meeting was held from March 24-26, 2010 in Chapel Hill, North Carolina. The first meeting was held in Nice, France in February 2009. The meeting attracted a wide range of attendees from experienced software developers who are contributing code to the open source iRODS software, to users with installations of varying sizes, to potential users wanting to learn more about iRODS. Users from academia, international and business organizations, and federal and state agencies were represented at the meeting.

With its advanced design and highly configurable and extensible architecture, iRODS is gaining increasing attention and users. The meeting attendance exceeded capacity with attendees from across the United States and as far away as France, Australia, and Taiwan, and participation from Japan by teleconference.

The call for participation encouraged submissions on a wide range of topics, including user applications of iRODS, interoperatibility and clients for accessing the iRODS data system, and integration with other data and repository systems. Contributions include papers, posters, and slides (available on the meeting website at https://www.irods.org/index.php/iRODS_User_Meetings).

The meeting began with sessions on an introduction to release 2.3 of iRODS, descriptions of how to create new micro-services and policies, a review of the unix shell command capabilities (icommands), and descriptions of the interactions (queries) on the iCAT metadata catalog.

A majority of the meeting was devoted to sessions for papers and presentations on user applications. The papers demonstrated the wide range of ways that communities have applied the iRODS framework. More than a dozen presentations and posters described use of iRODS in national data grids (e.g. Australian Research Collaboration Service), large-scale scientific research projects (e.g. KEK high energy physics data grid, French national computing center CC-IN2P3), institutional repositories (e.g. Carolina Digital Repository), and preservation environments (e.g. Taiwan National Archives).

Additional sessions were devoted to the development of new iRODS clients and integration of new capabilities into the iRODS framework. In particular, the JARGON Java I/O library was presented along with plans for integration with web services, development of a dropbox interface (iDrop), and support for digital library interfaces such as Islandora.

The final day of the meeting was dedicated to seeking community input into prioritizing new feature development, and identifying new types of interoperability mechanisms that are needed. A list was compiled of the current clients used to interact with an iRODS data grid. More than 35 existing clients were identified, ranging from Grid clients (GridFTP, JSAGA), file system interfaces (FUSE, PetaFS, webDAV), GUI interfaces (iExplore), digital library interfaces (Fedora, DSpace, Drupal, Islandora), workflow systems (Kepler, Taverna) and specialized interfaces such as URSpace for synchronizing local resources with iRODS. Interactions between collaborators are being supported through the Data Intensive Cyberinfrastructure Foundation, which supports the iRODS open source community.

The Data Intensive Cyberinfrastructure Foundation

iRODS 2011 User Group Meeting Announced

CI-BER project plans to store billions of electronic records

iRODS Version 2.4 Released

2010 iRODS User Group Meeting

The Data Intensive  Cyberinfrastructure Foundation