Aus Aifbportal
Wechseln zu:Navigation, Suche
LSDMA Logo.png

Large-Scale Data Management and Analysis

Contact: Fabian Rigoll

Project Status: completed


Large-Scale Data Management and Analysis within the scope of the program „ Scientific Computing “ were initiated in 2009 with first steps in view of a Large Scale Data Facility (LSDF). With an initial investment a prototyp data storage was built up and in it the first data was filed by the institute of Toxicology and Genetics. The preocupation with Large-Scale Data Management and Analysis will imply a main focus of the scientific works in the SCC during the next years. In view of Data Intensiv Computing and Data Analysis, which are key aspects of activity for the new founded Simulation Labs the LSDF is the ideal platform for research and development.

In KIT the institutes ITG, ANKA, IMK and IMF have announced an urgent request for storage and management of big data amounts. For the treatment of the scientific questions a close collaboration with the institutes IPE and IAI, as well as with industrial partners exists.

The following subjects are on the main research focus:

  • Long term archiving and integrity
  • Effective transfer and management of milliards of files (unified namespace)
  • Information Life Cycle of data
  • High capacity transport between storage classes (autom. workflow)
  • Development of interfaces to the LSDF and display of data structures in the browser
  • Data analysis and development of data analysis tools, efficient picture processing
  • Development of methods and tools related to Data Intensiv Computing

Beyond the focus of KIT Institutes there exists a cooperation with Bioquant at the University of Heidelberg. Within the scope of this cooperation the SCC received promotion of the MWK in Stuttgart in a scale of 6 PB disc storage (in 2010-2012) which will be also made available for the colleges of Baden-Wuerttemberg (State Concept). The LSDF in the SCC should be developed within the HGF to a central storage resource universally usable from all research facilities. (Quelle:

Involved Persons
Fabian Rigoll, Hartmut Schmeck


from: 1 Januar 2012
until: 31 Dezember 2016

Research Group

Efficient Algorithms

Area of Research

Energy Informatics

Publications Belonging to the Project
 - book
 - booklet
 - proceedings
 - phdthesis
 - techreport
 - deliverable
 - manual
 - misc
 - unpublished

Christopher J, M Gasthuber, A Giesler, M Hardt, J Meyer, A Prabhune, Fabian Rigoll, K Schwarz, A Streit
Progress in Multi-Disciplinary Data Life Cycle Management
Journal of Physics: Conference Series, 664, (3), pages 032018, 2015

↑ top

Christopher Jung, M. Gasthuber, A. Giesler, M. Hardt, J. Meyer, Fabian Rigoll, K. Schwarz, R. Stotzka, A. Streit
Optimization of data life cycles
Int. Conf. on Computing in High Energy and Nuclear Physics, 2013, Journal of Physics: Conference Series

Christopher Jung, Sören Fleischer, Martin Gasthuber, Andre Giesler, Marcus Hardt, Jörg Meyer, Fabian Rigoll, Rainer Stotzka, Max Fischer, Achim Streit
Advancing data management and analysis in different scientific disciplines
Proceedings of CHEP 2016, Proceedings of CHEP 2016, Oktober, 2016

↑ top

Fabian Rigoll, Hartmut Schmeck
A Concept for a User-oriented Energy Data Management System
In Christopher Jung, Jörg Meyer, Achim Streit, Helmholtz portfolio theme large-scale data management and analysis (LSDMA), pages 23-39, KIT Scientific Publishing, Karlsruhe, 2017

↑ top