Cloud Computing Testbed Chameleon Launches Third Phase with Focus on IoT and Reproducibility

$10 million NSF grant funds next four years of multi-institutional project

Since it launched in 2015, Chameleon has enabled systems and networking innovations by providing thousands of computer scientists with the bare metal access they need to conceptualize, assemble, and test new cloud computing approaches. 

Under a new four-year, $10 million grant from the National Science Foundation (NSF), the cloud computing testbed will further broaden its scope, adding new features for reproducibility, IoT and networking experimentation, and GPU computation to its core mission. This multi-institutional initiative is led by the University of Chicago (UChicago) in collaboration with the Renaissance Computing Institute (RENCI), Texas Advanced Computing Center (TACC), and Northwestern University.

“Chameleon is a scientific instrument for computer science systems research,” said Kate Keahey, senior computer scientist at Argonne National Laboratory and the Consortium for Advanced Science and Engineering (CASE) of the University of Chicago, and principal investigator of the Chameleon project. “Astronomers have telescopes, biologists have microscopes, and computer scientists have Chameleon.”

In its first five years, Chameleon has attracted more than 4,000 users from over 100 institutions, working on more than 500 different research and education projects. Scientists have used the testbed to study power management, operating systems, virtualization, high performance computing, distributed computing, networking, security, machine learning, and more. Educators have used Chameleon for cloud computing courses, allowing college and high school students to build their own cloud and learn the inner workings of the technology. 

The upcoming phase of Chameleon will further develop work already begun such as the popular CHameleon Infrastructure (CHI) that provides enhanced capabilities with the open source OpenStack project

The team will also broaden connections to other mission-specific testbeds, which will allow experimenters to implement core contributions of testbeds beyond Chameleon into their work. For example, Chameleon will expand capabilities for connecting IoT technologies by integrating with testbeds such as SAGE.

RENCI’s contributions to Chameleon in the third phase of funding will support this cross-testbed capability by further enabling experimentation with advanced programmable networking devices and accelerators. The RENCI team will also develop new options for software-defined networking that will allow compatibility with FABRIC, a currently-developing “everywhere programmable” nationwide instrument with large amounts of compute and storage, interconnected by high speed, dedicated optical links. 

“The planned additions to Chameleon will allow academic researchers to experiment with advanced programmable networks in a large-scale cloud environment,” said Paul Ruth, assistant director of network research and infrastructure at RENCI and co-PI on the Chameleon project. “We are excited to extend Chameleon’s cloud experiments into RENCI’s FABRIC testbed, which will facilitate larger, more diverse networking experiments.” 

Finally, the Chameleon team will also add expanded tools for reproducible research, and they will add new hardware and storage resources at the project’s two primary sites, UChicago and TACC, as well as at a supplemental Northwestern University site.

“Chameleon is a great example of how shared infrastructure with over 4,000 users can save the academic community time and money while catalyzing new research results,” said Deepankar Medhi, program director in the Computer & Information Sciences & Engineering Directorate (CISE) at the National Science Foundation. “NSF is pleased to fund Chameleon for four more years in order to extend the platform with new capabilities, thus allowing researchers to conduct new lines of research and students to learn newer technologies.”

To learn more about the testbed or begin experimenting on it today, visit chameleoncloud.org.

What to expect at the 2020 iRODS User Group Meeting

iRODS users and consortium members will gather virtually from June 9-12

The worldwide iRODS user community will connect online this week for the 12th Annual iRODS User Group Meeting – three days of learning, sharing of use cases, and discussions of new capabilities that have been added to iRODS in the last year.

The virtual event, sponsored by the University of Arizona, Cloudian, and RENCI, will be a collection of live talks with Q&A. An audience of nearly 300 participants representing dozens of academic, government, and commercial institutions is expected to join.

“The annual iRODS User Group Meeting has always opened our eyes to the impact of iRODS worldwide, and this year’s meeting will be no different,” says Jason Coposky, Executive Director at iRODS. “Although we are moving to a virtual platform, we intend to provide a similar experience to years past by ensuring there are plenty of opportunities for networking, discussion, and collaboration.”

Meeting attendees will learn about new updates such as hard links, direct streaming, and policy composition, according to Coposky. On June 12, the last day of the meeting, the Consortium team will run an iRODS Troubleshooting session, where participants can receive one-on-one help with an existing or planned iRODS installation or integration.

Last month, iRODS Consortium and RENCI announced the release of iRODS 4.2.8. A notable addition within the release was a new C++ rule engine plugin that provides an iRODS system the ability to convey hard links to its users. An iRODS system stores a hard link when replicas of two different iRODS data objects with different logical paths share a common physical path on the same host. When this occurs, metadata is added to both logical data objects for bookkeeping.

This year’s update to the iRODS S3 plugin shares the design and engineering underway to provide direct streaming into and out of S3-compatible storage. This rewrite uses the new iRODS IOStreams library and in-memory buffering to make efficient multi-part transfers.

With the addition of a continuation code to the rule engine plugin framework, iRODS users are now able to configure multiple policies to be invoked for any given policy enforcement point. The policy developers now have the ability to separate the policy enforcement points from the policy itself. Given this new approach, multiple policies can be configured together, or composed, without the need to touch the code.

As always with the annual UGM, in addition to general software updates, users will offer presentations about their organizations’ deployments of iRODS. This year’s meeting will feature 23 talks from users around the world. Among the use cases and deployments to be featured are:

  • SmartFarm data management, Agriculture Victoria. Data management challenges increase with large datasets generated with new sensing technologies. This requires the development of standardised, automated, on line, authenticated and verifiable standard processes for uploading data for storage and analytics on computing facilities. Agriculture Victoria undertakes research and development in animal and plant production, chemistry, spatial information, soil and water science. Working with iRODS, Agriculture Victoria are piloting new data management workflows of ‘SmartFarm’ data, and this talk will discuss lessons from small, medium, and high data Agriculture SmartFarm use cases using edge computing and collaborative data infrastructure and the flow on development of capability for AVR researchers.
  • Data management in autonomous driving projects, Aptiv. Aptiv is a global technology company that develops safer, greener, and more connected solutions that enable the future of mobility. The company deployed iRODS in production around 1.5 year ago, together with the start of the development phase of a big project on autonomous driving. The researchers will share how iRODS has assisted in tracking and migrating data between partners and within engineering groups responsible for data collection, manual and automatical analysis.
  • Building a national Research Data Management (RDM) infrastructure with iRODS in the Netherlands, SURF. In the Netherlands, many universities are looking at iRODS to support their researchers, as they recognize the powerful potential of the tool in two areas: support for secure cooperation, and support over the entire research data life cycle. SURF, a national organization providing IT support and infrastructure for universities, is now working closely together with six universities towards building a national RDM infrastructure based on iRODS. Researchers from SURF will share a case study for the use of iRODS, not for a specific research group, but for an entire nation to enhance the support of their researchers by working together on this iRODS based infrastructure.
  • Keeping pace with science: The CyVerse Data Store in 2020 and the Future, CyVerse / University of Arizona. CyVerse, hosted at the University of Arizona, provides a national cyberinfrastructure for life science research as well as training scientists in using such high performance computing resources.This talk will describe the current features of the CyVerse Data Store and plans for its evolution. Since its inception in 2010, the Data Store has leveraged the power and versatility of iRODS by continually extending the functionality of CyVerse’s cyberinfrastructure. These features include project-specific storage, offsite replication, third-party service and application integrations, several data access methods, event stream publishing for indexing, and optimizations for accessing large sets of small files.

Registration for the Virtual iRODS UGM will remain open throughout the week. See the registration page for details.


About the iRODS Consortium

The iRODS Consortium is a membership organization that supports the development of the integrated Rule-Oriented Data System (iRODS), free open source software for data virtualization, data discovery, workflow automation, and secure collaboration. The iRODS Consortium provides a production-ready iRODS distribution and iRODS training, professional integration services, and support. The world’s top researchers in life sciences, geosciences, and information management use iRODS to control their data. Learn more at irods.org.
The iRODS Consortium is administered by founding member RENCI, a research institute for applications of cyberinfrastructure at the University of North Carolina at Chapel Hill. For more information about RENCI, visit renci.org.

OpenIO joins iRODS Consortium

The iRODS Consortium, the foundation that leads development and support of the integrated Rule-Oriented Data System (iRODS) data management software, welcomes OpenIO as its newest Consortium member.

Read more

RENCI to help lead effort to make cancer research data more useful and accessible

The Renaissance Computing Institute (RENCI) at the University of North Carolina at Chapel Hill will collaborate on an $8.8 million, 3.5-year effort to make the volumes of data arising from cancer research more accessible, organized, and powerful. This contract was awarded by the Frederick National Laboratory for Cancer Research on behalf of the National Cancer Institute.

Read more

RENCI researchers spearhead $20 million project to test a reimagined Internet

Collaboration will establish a nationwide network infrastructure

The University of North Carolina at Chapel Hill will lead a $20 million project to create a platform for testing novel internet architectures that could enable a faster, more secure Internet.

With leadership from researchers at the Renaissance Computing Institute (RENCI), UNC-Chapel Hill and its partners will build a platform, called FABRIC, to provide a nationwide testbed for reimagining how data can be stored, computed and moved through shared infrastructure. FABRIC, funded by the National Science Foundation, will allow scientists to explore what a new Internet could look like at scale, and help determine the internet architecture of the future.

Read more

SUSE joins the iRODS Consortium

The iRODS Consortium, the foundation that leads development and support of the integrated Rule-Oriented Data System (iRODS) data management software, welcomes SUSE as its newest Consortium member.  

Read more
Tagged , , |

South Big Data Hub receives second round of NSF funding

$4 million will support continued innovation and problem-solving in the Southern data science community

The National Science Foundation (NSF) recently announced the second phase of funding for the regional Big Data Innovation Hubs (Hubs). Each of the Hubs will receive $4 million over four years for a total investment of $16 million.

Each Hub is located in one of the four U.S. Census regions (South, Northeast, Midwest, and West) and serves as a thought leader and convening force on social and economic challenges that are unique to the region by playing four key roles: (1) Accelerating public-private partnerships that break down barriers between industry, academia, and government, (2) Growing R&D communities that connect data scientists with domain scientists and practitioners, (3) Facilitating data sharing and shared cyberinfrastructure and services, and (4) Building data science capacity for education and workforce development.

Read more
Tagged , , |

University of Colorado Boulder Research Computing joins iRODS Consortium

The iRODS Consortium, the foundation that leads development and support of the integrated Rule-Oriented Data System (iRODS) data management software, welcomes University of Colorado Boulder (CU Boulder) Research Computing as its newest Consortium member.

CU Boulder Research Computing provides computing and data beyond the desktop to CU Boulder researchers and students. This includes large-scale computing resources, storage of research data, high-speed data transfer, data sharing support, and consultations in computational science and data management. 

Read more
Tagged , , |

Training, talks, and a hackathon bring users together for iRODS 2019 User Group Meeting

Seats are filling fast for international gathering of data management experts

Users of the integrated Rule Oriented Data System (iRODS) will gather at Utrecht University in the Netherlands June 26-27 for an annual opportunity to discuss iRODS-enabled applications and discoveries.

Read more
Tagged , , |

iRODS Consortium welcomes Maastricht University as newest member

Maastricht University, led by the efforts of DataHub Maastricht, which provides data management services to researchers from the university and academic hospital, has joined the iRODS Consortium, the foundation that leads development and support of the integrated Rule-Oriented Data System (iRODS). Maastricht is the fourth organization from the Netherlands to join the consortium, after Utrecht University, the SURF cooperative and the University of Groningen.

Read more
Tagged , , , |