node for reading files [request]

[clankill3r]

I'm in the last year of a studie interaction design. Here we are teached in processing for example, a java dialect great for turning data into graphics.
I always had a interest in modelling applications but this is more due gaming.
I started with xsi when the mod tool came in the days of half-life 2.
Since then i could never let go of it, the program is a master piece.
What i always wanted to do however is working with data files like csv (comma seperated values) or tsv (tab seperated values) in softimage.

I have been trying things and the String node in si doesn't work with returns for example.
Also there is a limit of 30000 characters for the string node.

Now i try to do it with jscript but what i would prefer is 2 descent ice nodes.

One is for creating a 2d array from a file structured like this:

Each row will be a new array, and its suited for files that have a lot of rows.

The ice node will look something like this:


Then another node would be able to create multiple arrays.
For data like this:

Where also the amount of rows can be huge but the amount of columns is fixed.

The node would look like this:


It would recognise the amount of columns and create that amount of array outputs.

I hope someone is interested in this and willing to help me.
I don't need it for my studie, it's just a unexplored thing for me and i do like to tests about data visualisation in softimage.
The node is meant to be released public and i would prefer open source as well but i leave that up to the programmer.
I'm able to pay a small amount of money but this won't be much since i have no job and i'm hoping for someone who does this for the love of softimage.

please let me know if your interested.

Moderator edit: I've overruled EricTRocks advice and moved this thread to "Plugins",
as the "Cooperation & Jobs" forum should kept as "clean" as possible, IMHO. - HB

[Daniel Brassard]

Interesting request although I see a few clarifications needed:

The 2D grid data parser will need some input from the user as to what type each columns of data is (string, float, integer, color, etc). That would help the parser in defining and checking the data for wrong input in the file, reserve space to each field and other data function.

Some form of header could be provided in the beginning of the file that define each field by name, type of data, field length etc. That header input could be used by the parser to define the output of the 2D grid data if that is what you want.

The user will have to provide a total size of the 2D grid data (row, columns) for memory management purpose. Depending of the file size, it could theoretically be bigger that what your system can handle and crash Soft. Some sort of warning or data block management may be required.

If you are going to have Tab value separated data file, do you want a 2D grid data for each tab versus row as originally requested?

Some thoughts for discussion.

[clankill3r]

The 2D grid data parser will need some input from the user as to what type each columns of data is (string, float, integer, color, etc). That would help the parser in defining and checking the data for wrong input in the file, reserve space to each field and other data function.

Yes, i would say passing a default value. Also because it's common to have lines with semicolons where there is no data for the colomn:
123;56;;;12;13
(happens on empty cells in excel)

You will be forced to give a default value, by that it will know it's type

For example force it to be a 2d array of type integer with a default value of -1.


For the 2nd one it would be nice if it works like this:

I select a file and choosed a delimiter.
It splits the first line in the file and find 3 columns by doing that (if another row would have more columns then it's just a bad structured file).

After that i set default values and by that force it to a type.

Some form of header could be provided in the beginning of the file that define each field by name, type of data, field length etc. That header input could be used by the parser to define the output of the 2D grid data if that is what you want.

I would prefer that things could be set in softimage. However you do mention a important point, often files allready do have a header line to describe what the column represents. Therefore it might be good to be able to skip that line with a boolean:


Or a start line and a end line:
-1 as end line will be till the end of the file.
This will also be good to only get a range

The user will have to provide a total size of the 2D grid data (row, columns) for memory management purpose. Depending of the file size, it could theoretically be bigger that what your system can handle and crash Soft. Some sort of warning or data block management may be required.

I think columns should be auto detect. For rows the start and end line above will be nice.
For the rest i never had really problems with big data, the biggest data file i worked with was 38.000 rows and around 50 columns and loading that in with java is a matter of a few seconds.

For unused columns it would be the best to set the data type to boolean even if it's a scalar for example. Converting the scalar to boolean will fail and there fore it will choose the default value you passed and that will be the boolean which doesn't tale much memory.

If you are going to have Tab value separated data file, do you want a 2D grid data for each tab versus row as originally requested?

I don't understand this question but maybe i answered.

[rray]

Nice idea, certainly would be cool.
A write array would complement this very well ;)
All rounded up with a "file exists" node

Maybe it would be enough/even more flexible to have a "get array from file column" node, which outputs only one specified column as an array.
Could quickly build a compound out of this which outputs all columns. Types could be picked in the node or autodetected as an option.

[luceric]

Also there is a limit of 30000 characters for the string node.

ice will always be re-parsing that string to convert it to arrays. that's very inefficient and unnecessary if the text file doesn't change all the time. it would be better to have text conversion outside of ice.

[clankill3r]

A write idea is more something for later

I like your "get array from file column" idea.
Here i made a example


And of course:


I also tested if it's possible to hold arrays of different types in a 2d array but that is not possible.

please keep providing input!

[rray]

File ops are ususally pretty fast/caching automatically in memory, so I think this might work very fast even without the caching attribute. Would need some testing.

[clankill3r]

This is how the other one will be atm, where each row is an array:

[Letterbox]

Also there is a limit of 30000 characters for the string node.

ice will always be re-parsing that string to convert it to arrays. that's very inefficient and unnecessary if the text file doesn't change all the time. it would be better to have text conversion outside of ice.

Like how?

[luceric]

softimage 2013 now has OM methods to create ICE attributes, so you could write a script that parses the text file and creates the attributes and then save the scene. then all the text parsing doesn't have to be redone all the time

[Letterbox]

softimage 2013 now has OM methods to create ICE attributes, so you could write a script that parses the text file and creates the attributes and then save the scene. then all the text parsing doesn't have to be redone all the time

So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance. [Hope your reading this guys ]

[clankill3r]

what is OM?
And is there any coder interested yet in helping me to get those nodes done?

[luceric]

So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance.

right you should never do work every frame that only needs to be done once ever. plus all that string parsing isn't multithreaded, it's just a waste of FPS.

I don't like how lame this is, but the way to find out that these OM methods exist is by looking at the What's New for 2013
http://download.autodesk.com/global/doc ... 915954.htm Maybe an sdk sample was made, I can't see it listed in the doc. These are used in ICE Crowd

[Letterbox]

So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance.

right you should never do work every frame that only needs to be done once ever. plus all that string parsing isn't multithreaded, it's just a waste of FPS.

I don't like how lame this is, but the way to find out that these OM methods exist is by looking at the What's New for 2013
http://download.autodesk.com/global/doc ... 915954.htm Maybe an sdk sample was made, I can't see it listed in the doc. These are used in ICE Crowd

Not lame.

Just thought I mention this, not sure about HOW ice would eval it. And its here if only for completeness. What if you have a compound that was in C that reads the file, and in that compound you add a boolean flag. That flag stating if it had done the read = set to 1, eg Self.FileRead=1. (which could be exposed too). Could you optimize out the file read operation out that way?

I never seen super clear exact example how NOT to get a branch to evaluate, so I wanted to mention this here as it seems very relevant, as people might need to do custom file formats.

IMPORTANT
PS I do think the python cvs for this example is the way to go, you dont want to rewrite parsers in C, if you dont have to.

[Hirazi Blue]

what is OM?
And is there any coder interested yet in helping me to get those nodes done?

OM is the abbreviation for the so-called "Object Model", one of the API's the SDK has on offer and specifically the one used predominantly when scripting (as opposed to the "C++ API", used, you guessed it, predominantly when coding in C++)

What seems to be the main message here, is, however, that if your file doesn't change its contents during the execution of the ICE Tree and it's basically "read once", the actual reading of the file could be done by scripting, maybe even on the custom PPG level (?), you wouldn't need a C++ node for that.

That doesn't solve your problem as of yet, but maybe clears the air a little...

[clankill3r]

well the downside of scripting is that it is not really user friendly for people that are into ice and not into scripting.
I allready tried with scripting and it's difference then i'm used to since it's of course more related to softimage.

How would you imagine the custom ppg to look like?