Import Particle data from an ASCII Text file

[ionside]

Sorry to open up an old thread here. Just wanted to say this has helped me quite a lot too and would like to say, thank you!

However! Hehe, I have a point cloud which is in the hundreds of millions of points (the client wants the detail). And the only way to get it across is to chop it up and convert, then stitch back up on SI.
How hard would it be to get this python script to chop up a file as it loads/converts it to say; every 40mb or every 400,000 points?

[ionside]

A friend, Niosop, modified the script, I hope that's OK? It splits a .pts file into a user defined number of chunks and writes an icecache file for each.

call it with something along the lines of "quick_parser.py filename.pts 0 50"
0 = the axis origin (0 = x, 1 = y, 2 = z).
50 = the number of chunks.

This quick_parser.py replaces the file of the same name.

Code: Select all

import os
import sys
import icecache  # Brad's API


def parse(filepath, coord, chunks, extents):
    for i in range(chunks):
        pos, color = list(), list()
        with open(filepath) as f:
            for line in f:
                try:
                    data = [float(x) for x in line.split(" ")]
                except:
                    continue
                    
                if(data.__len__() < 6):
                    continue
                    
                normalized_coord = (data[coord] - extents[0])/(extents[1] - extents[0])
                if normalized_coord < float(i)/chunks or normalized_coord > float(i+1)/chunks:
                    continue

                # filtering pointposition
                p = data[:3]
                p[1] *= -1  # negate y
                pos.append(p)
                # filtering colors
                c = [x / 255 for x in data[-3:]]
                c.append(1.0)  # adding alpha
                color.append(c)
        ic = icecache.icecache(len(pos))
        ic.addPointPosition(pos)
        ic.addColor("color", color)
        ic.write("%s-%d.icecache"%(os.path.splitext(filepath)[0], i))

def find_extents(filepath, coord):
    min_value = 1000000
    max_value = -1000000
    
    with open(filepath) as f:
        for line in f:
            try:
                data = [float(x) for x in line.split(" ")]
            except:
                continue
                
            if data.__len__() < 6:
                continue
                
            if data[coord] < min_value:
                min_value = data[coord]
            if data[coord] > max_value:
                max_value = data[coord]
                
    return [min_value, max_value]
    
    
if __name__ == "__main__":
    parse(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), find_extents(sys.argv[1], int(sys.argv[2])))

It's slow but good for large file/point sizes.

[csaez]

It's totally OK! thanks for sharing