Competition Overview
Self-driving cars (SDCs), leveraging sophisticated onboard cameras and sensors, are rapidly transitioning from concept to reality, demonstrating autonomous driving capabilities in diverse environments. This progress has sparked immense interest across numerous sectors. However, the comprehensive and rigorous testing of these complex autonomous systems, crucial for ensuring their safe and reliable operation in real-world scenarios, remains an area requiring significant attention and development, particularly concerning specialized Car Tester Tools.
Recognizing this critical need, the International Conference on Software Testing, Verification and Validation (ICST) and the Search-Based and Fuzz Testing (SBFT) workshop @ ICSE have joined forces to organize the SDC Testing Competition. This initiative is designed to galvanize the Software Testing Community, encouraging experts and innovators to focus their expertise on the rapidly evolving and vital field of SDC testing and the advancement of car tester tools. This collaborative call aims to minimize travel expenses for participants by offering a choice of conveniently located venues.
The competition is centered on regression testing methodologies applied to system tests of SDCs within simulated environments. Regression testing encompasses several key aspects, including test selection, prioritization, and minimization. For this particular competition, the primary focus is on test selection, a critical component in optimizing testing efficiency, especially when utilizing a car tester tool. Detailed information regarding test selection within the competition framework can be found on the competition platform repository.
Link to the competition platform: https://github.com/christianbirchler-org/sdc-testing-competition
Test Selection
Test Selection is a pivotal process in software testing, particularly when employing a car tester tool. It involves strategically choosing the most pertinent test cases from a comprehensive test suite in response to specific software modifications. In the context of simulation-based testing for SDCs, where test cases can be lengthy and resource-intensive, effective test selection is crucial. By using advanced car tester tools to selectively execute test cases, we can significantly reduce execution time while maintaining, or even enhancing, testing effectiveness.
Test Suite Selection [ ][ ][ ][ ] [x][ ][x][ ] [ ][ ][ ][ ] -> [x][ ][ ][x] -> Execution of the Selection -> Results [ ][ ][ ][ ] [ ][x][ ][ ]Aim
The central aim of this competition is to stimulate the development and submission of innovative and cost-effective test selectors – essentially, advanced car tester tools designed for autonomous vehicle systems. Following the submission deadline, all submitted tools will undergo rigorous evaluation by the competition organizers.
Participants whose submissions are accepted will be required to submit a concise paper detailing their car tester tool and its methodology to the competition organizers. Examples of such tool reports are available from SBFT 2024 https://sbft24.github.io/papers/.
Tool Submission
Participants have the flexibility to choose their preferred venue for competition: ICST 2025 or SBFT@ICSE 2025. It is important to note that participants in each venue will be evaluated and ranked independently, ensuring fair competition within each event. Detailed instructions regarding the tool submission process are comprehensively outlined on the competition platform, providing all necessary guidelines for submitting your car tester tool.
Tool Paper Submission
The deadline for camera-ready submissions is January 10, 2024 (AoE). Participants are encouraged to share draft versions of their papers with the organizers for review before January 3rd, allowing for valuable feedback and improvement opportunities.
Following acceptance notification, the publisher will contact participants with specific instructions for preparing and submitting the camera-ready versions of their papers. It is crucial to adhere to the strict page limits, which are set at 2 pages for tool papers. Please ensure that the list of authors (including names, emails, affiliations, and order) and the paper title remain unchanged. Any necessary corrections should be promptly communicated to the organizers via email.
This tool competition builds upon a foundation of prior research and development in the field. Therefore, when preparing your paper on your car tester tool, please consider referencing the following publications related to test selection for SDCs:
@inproceedings{sdc-scissor, title={Cost-effective simulation-based test selection in self-driving cars software with SDC-Scissor}, author={Birchler, Christian and Ganz, Nicolas and Khatiri, Sajad and Gambi, Alessio and Panichella, Sebastiano}, booktitle={2022 IEEE international conference on software analysis, evolution and reengineering (SANER)}, pages={164--168}, year={2022}, organization={IEEE} } @article{ml-test-selection, title={Machine learning-based test selection for simulation-based testing of self-driving cars software}, author={Birchler, Christian and Khatiri, Sajad and Bosshard, Bill and Gambi, Alessio and Panichella, Sebastiano}, journal={Empirical Software Engineering}, volume={28}, number={3}, pages={71}, year={2023}, publisher={Springer US New York} } @inproceedings{teaser, title={TEASER: Simulation-Based CAN Bus Regression Testing for Self-Driving Cars Software}, author={Birchler, Christian and Rohrbach, Cyrill and Kim, Hyeongkyun and Gambi, Alessio and Liu, Tianhai and Horneber, Jens and Kehrer, Timo and Panichella, Sebastiano}, booktitle={2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)}, pages={2058--2061}, year={2023}, organization={IEEE} }For submissions to ICST 2025, please include the following citation:
@inproceedings{ICST-SDC2025, author={Birchler, Christian and Klikovits, Stefan and Fazzini, Mattia and Panichella, Sebastiano}, title={{ICST} Tool Competition 2025 - Self-Driving Car Testing Track}, booktitle={{IEEE/ACM} International Conference on Software Testing, Verification and Validation, ICST 2025}, year={2025} }For submissions to SBFT 2025, include this reference:
@inproceedings{SBFT-SDC2025, author = {{Birchler, Christian and Klikovits, Stefan and Fazzini, Mattia and Panichella, Sebastiano}, title = {{SBFT} Tool Competition 2025 - CPS-SDC Regression Testing Track}, booktitle = {{IEEE/ACM} International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2025}, year = {2025} }A presentation by at least one author at the respective conference is mandatory for each accepted paper. Registration details can be found on the ICST conference website, ensuring your participation in showcasing your innovative car tester tool.
Competition Platform
The GitHub platform serves as the central hub for all competition guidelines, detailed descriptions, and a comprehensive Q&A section. The competition platform is publicly accessible at: https://github.com/christianbirchler-org/sdc-testing-competition. Participants are encouraged to utilize the GitHub platform to discuss any issues, queries, or topics related to the competition directly, fostering a collaborative environment for car tester tool development.
Sponsors
We extend our sincere gratitude to BeamNG GmbH for their generous sponsorship and support of this conference and the Self-Driving Car Testing Competition track, enabling advancements in car tester tools and SDC technology.
