Got it. I know Lidartools (which LAStools is dependent on) was not ported for QGIS 3 so that what I was getting at. 10M points isn’t all that big so if you have at least 16GB of RAM you shouldn’t have any problems, but considering you are having problems with a 500 point subset then we know that combined with the scaling issue that it’s probably not your plug-ins. Have you tried downloading and importing an XYZ version?
You might try PDAL.
https://grass.osgeo.org/grass78/manuals/v.in.pdal.html