BR-SP-SPRACE cluster: A Tier 2 of the WLCG

The SPRACE cluster is a Tier 2 computing center of the Compact Muon Solenoid (CMS) experiment, one of the two large general-purpose particle physics detectors of the Large Hadron Collider (LHC) at CERN. It was funded by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) through thematic projects nº03/04519-9 (2003-2008) and nº08/02799-8 (2008-2013).

The SPRACE cluster began its operations in 2004, processing DZero/Fermilab data. In 2005 SPRACE became an Open Science Grid (OSG) official site and in 2006 joined the CMS experiment. In 2009 SPRACE signed the WLCG Memorandum of Understanding (MoU), becoming an official WLCG Tier 2 facility (BR-SP-SPRACE), the only official Latin America WLCG Tier 2 site.

Currently it has 25.20K HEPSPEC06 of processing power in 128 servers with 1,792 CPU cores, and 2,000 TB of storage (2.8 PB raw) in 17 storage servers, all with 10GbE or 40GbE connection. A more detailed description of the computing and storage resources of the SPRACE cluster can be found here.

GridUNESP – a SPRACE Spin-off

GridUNESP is a multi-campus Grid Computing infrastructure for the São Paulo State University (UNESP). It was the first campus Grid in Latin America built to provide processing power and storage resources to the researchers of UNESP.

GridUNESP is a spin-off of the SPRACE project and follows the tier topology of the WLCG. It has a Tier 0 site located in the city of São Paulo and other six Tier 1 sites distributed across the state. There are 370 servers (spread over 7 sites) with 2,944 CPU cores, providing 33.3 teraflops of computing power, and 160 TB of storage in 10 storage servers (also spread over 7 sites).

GridUNESP is part of the Open Science Grid cyberinfrastructure, being the only OSG Virtual Organization (VO) outside the USA. Since the system came to life, in Jan 2010, it produced more than 5.2 million simulations, corresponding to 180 million CPU hours.

Currently the project serves more than 80 research projects with more than 450 users.

Supercomputing Conference and Bandwidth Challenges

In 2016, at the SuperComputing Conference (SC16), SPRACE, in collaboration with the Academic Network of São Paulo (ANSP), Americas Pathways (AmPATH) and the California Institute of Technology (Caltech), set a new data transmission record between the Northern-Southern hemispheres by using new international 100 Gbps links. Data transfer reached 97,56 Gbps and averaged 96,56 Gbps during an hour of transmission. Thanks to SPRACE the São Paulo State University (Unesp) is currently the only academic institution in Latin America capable of transferring data at 100 Gbps to the United States.

This was the third time the SPRACE team set the data transfer record between hemispheres. In 2004 transmissions reached 1 Gbps and, in 2009, 16,5 Gbps. SPRACE has been participating in these SuperComputing demonstrations and bandwidth challenges, in collaboration with Caltech, for over 12 years.