Abstract
In this paper we describe the design principles, implementation choices and general challenges we
encountered in the creation of a data management infrastructure for recording data streams from test vehicles, robots and other platforms. The trigger for this data management infrastructure project was twofold: First from the proper setup of new test cars equipped with many sensors, delivering high bandwidth data recordings and second from achieving organized storage of such
recordings for the development and testing of intelligent systems operating on the data. After the clearly stated demand of such a data management system from different divisions of our company, we, step by step, conceived it as a very general data management platform targeting different projects with different recording formats and platforms. Recording data from different projects have
systematic commonalities, for instance most use time series of data, often from similar type sensors with similar information. However considerable differences exist with respect to data organization in recording sessions, stream formatting or coverage of specific situation/event or environment conditions. Our data management infrastructure targets to support different needs in the data management work-flow. Facilitating recordings visualization, search, inspection, annotation of
events/entities present in the data and offline access is among our main targets. Our approach is first to centralize storage of recordings, avoiding proliferation of copies of them in our network.
We give GUI and programmatic access allowing both tool-based-manual-annotation processes as well as automated processes using AI/deep-learning methods. Subsequently we support extraction of events or meta-information from recordings, storing them in a database.. Our infrastructure enables then an efficient search over extracted information for exporting relevant recording segments, used for the creation of automotive or robotics intelligent systems.
encountered in the creation of a data management infrastructure for recording data streams from test vehicles, robots and other platforms. The trigger for this data management infrastructure project was twofold: First from the proper setup of new test cars equipped with many sensors, delivering high bandwidth data recordings and second from achieving organized storage of such
recordings for the development and testing of intelligent systems operating on the data. After the clearly stated demand of such a data management system from different divisions of our company, we, step by step, conceived it as a very general data management platform targeting different projects with different recording formats and platforms. Recording data from different projects have
systematic commonalities, for instance most use time series of data, often from similar type sensors with similar information. However considerable differences exist with respect to data organization in recording sessions, stream formatting or coverage of specific situation/event or environment conditions. Our data management infrastructure targets to support different needs in the data management work-flow. Facilitating recordings visualization, search, inspection, annotation of
events/entities present in the data and offline access is among our main targets. Our approach is first to centralize storage of recordings, avoiding proliferation of copies of them in our network.
We give GUI and programmatic access allowing both tool-based-manual-annotation processes as well as automated processes using AI/deep-learning methods. Subsequently we support extraction of events or meta-information from recordings, storing them in a database.. Our infrastructure enables then an efficient search over extracted information for exporting relevant recording segments, used for the creation of automotive or robotics intelligent systems.
| Original language | English |
|---|---|
| Pages (from-to) | 1-9 |
| Number of pages | 9 |
| Journal | The Japanese Society for Artificial Intelligence |
| Volume | 33 |
| Issue number | 1 |
| State | Published - Jan 2018 |
Keywords
- big-data
- data recordings
- multi-sensor systems
- data management
- intelligent systems
- AI
- automotive
- robotics