Evaluate and share your results (not available currently)

WHU3D Benchmark

MLS Instance Segmentation

Methods MUCov MWCov Pre Rec F1 oAcc mAcc mIoU
JSIS3D 0.162 0.194 0.104 0.108 0.106 0.681 0.305 0.225
SoftGroup 0.297 0.319 0.099 0.289 0.115 0.741 0.448 0.250
SPFormer 0.301 0.356 0.220 0.334 0.251 - - -
ISBNet 0.304 0.332 0.131 0.295 0.167 0.685 0.467 0.281

ALS Instance Segmentation

Methods MUCov MWCov Pre Rec F1 oAcc mAcc mIoU
JSIS3D 0.125 0.179 0.077 0.080 0.078 0.579 0.393 0.280
SoftGroup 0.295 0.314 0.131 0.259 0.173 0.597 0.518 0.289
SPFormer 0.420 0.458 0.110 0.378 0.151 - - -
ISBNet 0.328 0.352 0.226 0.315 0.228 0.764 0.643 0.530

MLS Semantic Segmentation

<
- KPConv Minkowski PointTransformer OctFormer
oAcc 0.840 0.837 0.829 0.849
mAcc 0.518 0.649 0.644 0.641
IoUKPConv Minkowski PointTransformer OctFormer
Mean 0.428 0.536 0.524 0.528
Tree 0.875 0.869 0.860 0.884
Nd.way 0.500 0.447 0.434 0.522
Building 0.839 0.820 0.822 0.803
Box 0.554 0.688 0.548 0.552
Light 0.375 0.504 0.476 0.501
Ele.pole 0.000 0.000 0.000 0.017
Mun.pole 0.000 0.000 0.208 0.110
Low.veg. 0.320 0.332 0.325 0.301
Board 0.272 0.462 0.415 0.363
Drive Way 0.854 0.834 0.824 0.869
Mark 0.295 0.376 0.332 0.408
Vehicle 0.896 0.870 0.820 0.872
Pedestrian 0.809 0.782 0.677 0.793
Traff.light 0.000 0.529 0.587 0.632
Detector 0.000 0.156 0.273 0.188
Fence 0.544 0.627 0.609 0.576
Wire 0.000 0.721 0.665 0.568
Pole 0.575 0.624 0.562 0.553

ALS Semantic Segmentation

- KPConv Minkowski PointTransformer OctFormer
oAcc 0.772 0.781 0.792 0.743
mAcc 0.625 0.633 0.684 0.538
loU KPConv Minkowski PointTransformer OctFormer
Mean 0.524 0.529 0.572 0.445
Ground 0.640 0.667 0.667 0.629
Building 0.799 0.845 0.862 0.800
Tree 0.302 0.397 0.392 0.110
Veg. 0.693 0.696 0.700 0.665
Low.veg. 0.458 0.417 0.439 0.272
Light 0.000 0.000 0.278 0.000
Electric 0.774 0.680 0.667 0.639

Upload Your Results

Users could also upload your results to our benchmark page. The structure of the file should be: 

                    
                        mls.zip
                        └── h5
                            ├── [scene_1].h5
                            ├── [scene_2].h5
                            └── [scene_*].h5
Each file contains a "labels" key in it, with semantic and instance labels corresponding with coordinates.

© 2023 WHU3D. All rights reserved.