CARLA RAI Challenge > RAI Leaderboard

RAI Leaderboard

The CARLA RAI Autonomous Driving challenge requires that agents navigate predefined routes with various real-world traffic scenarios. Agents, initialized at a designated starting point, are directed to drive to a specified destination using route information presented in GPS-style coordinates or route instructions. The primary objective of the CARLA RAI AD leaderboard is to assess how robust the agents are to different challenging naturally occuring traffic conditions, how much they impact environmental sustainability, and whether transparency/interpretability measures were implemented.

ChallengeTeaser

Scenarios

Agents will face multiple traffic scenarios based on a subset of scenarios from CARLA Scenario Runner. Scenarios involves events as: .

Lane Following/Change.
Negotiations at traffic intersections.
Negotiations at roundabouts.
Handling traffic lights and traffic signs.
Coping with pedestrians, cyclists, and other elements.

Participation Modalities

Unlike the original CARLA AD challenge, our leaderboard only offers one participation modality, which is the SENSORS modality. Agents will receive a high-level route description indicating the key points that the path has to follow in order to reach the destination. The route is represented as a list of tuples and has two variations.

For the first case, the first element of the tuple contains a waypoint, expressed as a latitude, a longitude, and a z component.

[({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002271601998707}, RoadOption.LEFT),
 ({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002709765148996}, RoadOption.RIGHT),
 ...
 ({'z': 0.0, 'lat': 48.99822679980298, 'lon': 8.002735250105061}, RoadOption.STRAIGHT)]

The second case is very similar to the previous one, but instead of using GPS coordinates, the route is expressed in world coordinates instead.

[({'x': 153.7, 'y': 15.6, 'z': 0.0}, RoadOption.LEFT),
 ({'x': 148.9, 'y': 67.8, 'z': 0.0}, RoadOption.RIGHT),
 ...
 ({'x': 180.7, 'y': 45.1, 'z': 1.2}, RoadOption.STRAIGHT)]

The distance between two consecutive waypoints could be up to hundreds of meters. Do not rely on these as your principal mechanism to navigate the environment.

The second element contains a high-level command. The set of available high-level commands is:

RoadOption.CHANGELANELEFT: Move one lane to the left.
RoadOption.CHANGELANERIGHT: Move one lane to the right.
RoadOption.LANEFOLLOW: Continue in the current lane.
RoadOption.LEFT: Turn left at the intersection.
RoadOption.RIGHT: Turn right at the intersection.
RoadOption.STRAIGHT: Keep straight at the intersection.

There might be cases where the semantics of left and right is ambiguous. In order to disambiguate these situations, you could consider the GPS position of the next waypoints.

You are not allowed to make use of any privilege information offered by the CARLA simulator, including planners or any type of ground truth. Submissions using these features will be rejected and teams will be banned from the platform.

Sensors Available

Agents can request access to the following sensors:

GNSS	IMU	LIDAR	RGB Camera	Speedometer
sensor.other.gnss	sensor.other.imu	sensor.lidar.ray_cast	sensor.camera.rgb	sensor.other.speedometer
0-1 units	0-1 units	0-1 units	0-4 units	0-1 units
GPS sensor returning geo location data.	6-axis Inertial Measurement Unit.	Velodyne 64 LIDAR	Regular camera that captures images.	Pseudosensor that provides an approximation of your linear velocity.

Evaluation and Metrics

For this leaderboard we have selected a set of metrics that help understand different aspects of driving. The specific metrics are as follows:

Regular Driving Score

The driving score $D_{i} = R_{i} P_{i}$ is the product of the route completion and the infraction penalty. Here $R_{i}$ is the percentage of completion of the $i-th$ route and $P_{i}$ is the infraction penalty incurred during the $i-th$ route.

Infraction penalty: We track different types of infractions (e.g., collision with a vehicle, running a red light, etc.) in which the agent was involved. The infraction penalty score aggregates all of the infractions as a geometric series along a particular route. Agents start with an ideal 1.0 base penalty score, which is reduced each time an infraction is committed. Infraction penalty $P_{i}$ was computed as: $\prod\limits_{l}^{|{W}|} {(P_{l})}^{\left|{W_l}\right|}$, where $W$ is a set of infractions, $l$ is an infraction type in $W$, and $|W_l|$ is the number of $W_l$ infractions that occurred, and $P_l$ is the cost for infraction type $l$.

Robustness Driving Score

Each route would be run more than once depending on which type of traffic disturbances are introduced, e.g., if the traffic disturbance is 'weather' and there are 3 weather types, then the route would be run 3 times corresponding to the 3 different weather condition. When a route has been run for the desired number of times:

(i) The driving scores (i.e., after penalties have been applied) for the runs are grouped based on the type of traffic disturbances introduced. For example, 'weather' driving scores […], 'camera noise' driving scores […], …
(ii) final score for each type of traffic disturbance is obtained by selecting the minimum score from each group. For example, if a route was run in $n$ different weather conditions, the driving score for 'weather' would be the least driving score obtained after running the agent in the $n$ different weather conditions. If there were more than one routes, we will use the score from the route that produced the least score for the selected traffic disturbance type.
(iii) robustness ratio is computed as the ratio of the driving score obtained when the disturbance is introduced and the driving score obtained in the normal/regular driving case. The normal driving case is the case without artificially introduced disturbances.
(iv) a robustness driving score for which agents are ranked is computed as by taking the mean of the final scores (from step ii.) for the different traffic disturbance types.

See paper for further details.

Environmental Sustainability

For each run we estimate the amount of $\text{CO}_2$ Eq emitted as a result of running the agent. Note that only the system process running the agent is tracked. We provide the average $\text{CO}_2$ Eq emission per run as well as the average $\text{CO}_2$ Eq emission per seconds.

See paper for further details.

Transparency

As it is currently difficult to automatically assess weather an agent is interpretable or not, in this version of CARLA RAI Challenge, we are only using the qualitative responses provided by the participants (including evidences from their submissions) regarding the transparency measures/considerations they made while developing their agents.

See paper for further details.

Infractions and shutdown events (Contd.)

The CARLA leaderboard offers individual metrics for a series of infractions. Each of these has a penalty coefficient that will be applied everytime it happens. Ordered by severity, the infractions are the following.

Collisions with pedestrians — 0.50.
Collisions with other vehicles — 0.60.
Collisions with static elements — 0.65.
Running a red light — 0.70.
Running a stop sign — 0.80.

Some scenarios feature behaviors that can block the ego-vehicle indefinitely. These scenarios will have a timeout of 4 minutes after which the ego-vehicle will be released to continue the route. However, a penalty is applied when the time limit is breached:

Scenario timeout — 0.7

The agent is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will be compared with the speed of nearby vehicles. Failure to maintain a suitable speed will result in a penalty. The penalty applied is dependent on the magnitude of the speed difference, up to the following value:

Failure to maintain minimum speed — 0.7

The agent should yield to emergency vehicles coming from behind. Failure to allow the emergency vehicle to pass will incur a penalty:

Failure to yield to emergency vehicle — 0.7

Besides these, there is one additional infraction which has no coefficient, and instead affects the computation of the route completion (R_i).

Off-road driving — If an agent drives off-road, that percentage of the route will not be considered towards the computation of the route completion score. Additionally, some events will interrupt the simulation, preventing the agent to continue. In these cases, the route which is being simulated will be shut down, and the leaderboard will move onto the next one, triggering it normally.
Route deviation — If an agent deviates more than 30 meters from the assigned route.
Agent blocked — If an agent doesn’t take any actions for 180 simulation seconds.
Simulation timeout — If no client-server communication can be established in 60 seconds.
Route timeout — If the simulation of a route takes too long to finish.

Each time any of the above happens, several details are recorded, which will be displayed as a list for you to see at the route’s individual metrics. Below is an example of a route where the agent both run a red light and deviated from the route.

"infractions": {
  "Collisions with layout": [],
  "Collisions with pedestrians": [],
  "Collisions with vehicles": [],
  "Red lights infractions": [
        "Agent ran a red light 203 at (x=341.25, y=209.1, z=0.104)"
  ],
  "Stop sign infractions": [],
  "Off-road infractions": [],
  "Min speed infractions": [],
  "Yield to emergency vehicle infractions": [],
  "Scenario timeouts": [],
  "Route deviations": [
        "Agent deviated from the route at (x=95.92, y=165.673, z=0.138)"
  ],
  "Agent blocked": [],
  "Route timeouts": []
}