node for reading files [request]
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
node for reading files [request]
I'm in the last year of a studie interaction design. Here we are teached in processing for example, a java dialect great for turning data into graphics.
I always had a interest in modelling applications but this is more due gaming.
I started with xsi when the mod tool came in the days of half-life 2.
Since then i could never let go of it, the program is a master piece.
What i always wanted to do however is working with data files like csv (comma seperated values) or tsv (tab seperated values) in softimage.
I have been trying things and the String node in si doesn't work with returns for example.
Also there is a limit of 30000 characters for the string node.
Now i try to do it with jscript but what i would prefer is 2 descent ice nodes.
One is for creating a 2d array from a file structured like this:
Each row will be a new array, and its suited for files that have a lot of rows.
The ice node will look something like this:
Then another node would be able to create multiple arrays.
For data like this:
Where also the amount of rows can be huge but the amount of columns is fixed.
The node would look like this:
It would recognise the amount of columns and create that amount of array outputs.
I hope someone is interested in this and willing to help me.
I don't need it for my studie, it's just a unexplored thing for me and i do like to tests about data visualisation in softimage.
The node is meant to be released public and i would prefer open source as well but i leave that up to the programmer.
I'm able to pay a small amount of money but this won't be much since i have no job and i'm hoping for someone who does this for the love of softimage.
please let me know if your interested.
Moderator edit: I've overruled EricTRocks advice and moved this thread to "Plugins",
as the "Cooperation & Jobs" forum should kept as "clean" as possible, IMHO. - HB
I always had a interest in modelling applications but this is more due gaming.
I started with xsi when the mod tool came in the days of half-life 2.
Since then i could never let go of it, the program is a master piece.
What i always wanted to do however is working with data files like csv (comma seperated values) or tsv (tab seperated values) in softimage.
I have been trying things and the String node in si doesn't work with returns for example.
Also there is a limit of 30000 characters for the string node.
Now i try to do it with jscript but what i would prefer is 2 descent ice nodes.
One is for creating a 2d array from a file structured like this:
Each row will be a new array, and its suited for files that have a lot of rows.
The ice node will look something like this:
Then another node would be able to create multiple arrays.
For data like this:
Where also the amount of rows can be huge but the amount of columns is fixed.
The node would look like this:
It would recognise the amount of columns and create that amount of array outputs.
I hope someone is interested in this and willing to help me.
I don't need it for my studie, it's just a unexplored thing for me and i do like to tests about data visualisation in softimage.
The node is meant to be released public and i would prefer open source as well but i leave that up to the programmer.
I'm able to pay a small amount of money but this won't be much since i have no job and i'm hoping for someone who does this for the love of softimage.
please let me know if your interested.
Moderator edit: I've overruled EricTRocks advice and moved this thread to "Plugins",
as the "Cooperation & Jobs" forum should kept as "clean" as possible, IMHO. - HB
- Daniel Brassard
- Posts: 878
- Joined: 18 Mar 2010, 23:38
- Location: St. Thomas, Ontario
- Contact:
Re: node for reading files [request]
Interesting request although I see a few clarifications needed:
The 2D grid data parser will need some input from the user as to what type each columns of data is (string, float, integer, color, etc). That would help the parser in defining and checking the data for wrong input in the file, reserve space to each field and other data function.
Some form of header could be provided in the beginning of the file that define each field by name, type of data, field length etc. That header input could be used by the parser to define the output of the 2D grid data if that is what you want.
The user will have to provide a total size of the 2D grid data (row, columns) for memory management purpose. Depending of the file size, it could theoretically be bigger that what your system can handle and crash Soft. Some sort of warning or data block management may be required.
If you are going to have Tab value separated data file, do you want a 2D grid data for each tab versus row as originally requested?
Some thoughts for discussion.
The 2D grid data parser will need some input from the user as to what type each columns of data is (string, float, integer, color, etc). That would help the parser in defining and checking the data for wrong input in the file, reserve space to each field and other data function.
Some form of header could be provided in the beginning of the file that define each field by name, type of data, field length etc. That header input could be used by the parser to define the output of the 2D grid data if that is what you want.
The user will have to provide a total size of the 2D grid data (row, columns) for memory management purpose. Depending of the file size, it could theoretically be bigger that what your system can handle and crash Soft. Some sort of warning or data block management may be required.
If you are going to have Tab value separated data file, do you want a 2D grid data for each tab versus row as originally requested?
Some thoughts for discussion.
$ifndef "Softimage"
set "Softimage" "true"
$endif
set "Softimage" "true"
$endif
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
Re: node for reading files [request]
Yes, i would say passing a default value. Also because it's common to have lines with semicolons where there is no data for the colomn:The 2D grid data parser will need some input from the user as to what type each columns of data is (string, float, integer, color, etc). That would help the parser in defining and checking the data for wrong input in the file, reserve space to each field and other data function.
123;56;;;12;13
(happens on empty cells in excel)
You will be forced to give a default value, by that it will know it's type
For example force it to be a 2d array of type integer with a default value of -1.
For the 2nd one it would be nice if it works like this:
I select a file and choosed a delimiter.
It splits the first line in the file and find 3 columns by doing that (if another row would have more columns then it's just a bad structured file).
After that i set default values and by that force it to a type.
I would prefer that things could be set in softimage. However you do mention a important point, often files allready do have a header line to describe what the column represents. Therefore it might be good to be able to skip that line with a boolean:Some form of header could be provided in the beginning of the file that define each field by name, type of data, field length etc. That header input could be used by the parser to define the output of the 2D grid data if that is what you want.
Or a start line and a end line:
-1 as end line will be till the end of the file.
This will also be good to only get a range
I think columns should be auto detect. For rows the start and end line above will be nice.The user will have to provide a total size of the 2D grid data (row, columns) for memory management purpose. Depending of the file size, it could theoretically be bigger that what your system can handle and crash Soft. Some sort of warning or data block management may be required.
For the rest i never had really problems with big data, the biggest data file i worked with was 38.000 rows and around 50 columns and loading that in with java is a matter of a few seconds.
For unused columns it would be the best to set the data type to boolean even if it's a scalar for example. Converting the scalar to boolean will fail and there fore it will choose the default value you passed and that will be the boolean which doesn't tale much memory.
I don't understand this question but maybe i answered.If you are going to have Tab value separated data file, do you want a 2D grid data for each tab versus row as originally requested?
Re: node for reading files [request]
Nice idea, certainly would be cool.
A write array would complement this very well ;)
All rounded up with a "file exists" node
Maybe it would be enough/even more flexible to have a "get array from file column" node, which outputs only one specified column as an array.
Could quickly build a compound out of this which outputs all columns. Types could be picked in the node or autodetected as an option.
A write array would complement this very well ;)
All rounded up with a "file exists" node
Maybe it would be enough/even more flexible to have a "get array from file column" node, which outputs only one specified column as an array.
Could quickly build a compound out of this which outputs all columns. Types could be picked in the node or autodetected as an option.
softimage resources section updated Jan 5th 2024
Re: node for reading files [request]
ice will always be re-parsing that string to convert it to arrays. that's very inefficient and unnecessary if the text file doesn't change all the time. it would be better to have text conversion outside of ice.Also there is a limit of 30000 characters for the string node.
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
Re: node for reading files [request]
A write idea is more something for later
I like your "get array from file column" idea.
Here i made a example
And of course:
I also tested if it's possible to hold arrays of different types in a 2d array but that is not possible.
please keep providing input!
I like your "get array from file column" idea.
Here i made a example
And of course:
I also tested if it's possible to hold arrays of different types in a 2d array but that is not possible.
please keep providing input!
Re: node for reading files [request]
File ops are ususally pretty fast/caching automatically in memory, so I think this might work very fast even without the caching attribute. Would need some testing.
softimage resources section updated Jan 5th 2024
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
Re: node for reading files [request]
This is how the other one will be atm, where each row is an array:
Re: node for reading files [request]
Like how?luceric wrote:ice will always be re-parsing that string to convert it to arrays. that's very inefficient and unnecessary if the text file doesn't change all the time. it would be better to have text conversion outside of ice.Also there is a limit of 30000 characters for the string node.
Re: node for reading files [request]
softimage 2013 now has OM methods to create ICE attributes, so you could write a script that parses the text file and creates the attributes and then save the scene. then all the text parsing doesn't have to be redone all the time
Re: node for reading files [request]
luceric wrote:softimage 2013 now has OM methods to create ICE attributes, so you could write a script that parses the text file and creates the attributes and then save the scene. then all the text parsing doesn't have to be redone all the time
So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance. [Hope your reading this guys ]
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
Re: node for reading files [request]
what is OM?
And is there any coder interested yet in helping me to get those nodes done?
And is there any coder interested yet in helping me to get those nodes done?
Re: node for reading files [request]
right you should never do work every frame that only needs to be done once ever. plus all that string parsing isn't multithreaded, it's just a waste of FPS.Letterbox wrote:So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance.
I don't like how lame this is, but the way to find out that these OM methods exist is by looking at the What's New for 2013
http://download.autodesk.com/global/doc ... 915954.htm Maybe an sdk sample was made, I can't see it listed in the doc. These are used in ICE Crowd
Re: node for reading files [request]
Not lame.luceric wrote:right you should never do work every frame that only needs to be done once ever. plus all that string parsing isn't multithreaded, it's just a waste of FPS.Letterbox wrote:So correct me if I'm wrong, the guys should use pythons CSV functions connect those into the new 2013 OM methods (in the sdk help?), as doing it the way they thought or by creating/using C++ file nodes with ICE, will cause re-evaluations, and cripple performance.
I don't like how lame this is, but the way to find out that these OM methods exist is by looking at the What's New for 2013
http://download.autodesk.com/global/doc ... 915954.htm Maybe an sdk sample was made, I can't see it listed in the doc. These are used in ICE Crowd
Just thought I mention this, not sure about HOW ice would eval it. And its here if only for completeness. What if you have a compound that was in C that reads the file, and in that compound you add a boolean flag. That flag stating if it had done the read = set to 1, eg Self.FileRead=1. (which could be exposed too). Could you optimize out the file read operation out that way?
I never seen super clear exact example how NOT to get a branch to evaluate, so I wanted to mention this here as it seems very relevant, as people might need to do custom file formats.
IMPORTANT
PS I do think the python cvs for this example is the way to go, you dont want to rewrite parsers in C, if you dont have to.
- Hirazi Blue
- Administrator
- Posts: 5107
- Joined: 04 Jun 2009, 12:15
Re: node for reading files [request]
OM is the abbreviation for the so-called "Object Model", one of the API's the SDK has on offer and specifically the one used predominantly when scripting (as opposed to the "C++ API", used, you guessed it, predominantly when coding in C++)clankill3r wrote:what is OM?
And is there any coder interested yet in helping me to get those nodes done?
What seems to be the main message here, is, however, that if your file doesn't change its contents during the execution of the ICE Tree and it's basically "read once", the actual reading of the file could be done by scripting, maybe even on the custom PPG level (?), you wouldn't need a C++ node for that.
That doesn't solve your problem as of yet, but maybe clears the air a little...
Stay safe, sane & healthy!
-
- Posts: 131
- Joined: 21 Oct 2012, 15:36
Re: node for reading files [request]
well the downside of scripting is that it is not really user friendly for people that are into ice and not into scripting.
I allready tried with scripting and it's difference then i'm used to since it's of course more related to softimage.
How would you imagine the custom ppg to look like?
I allready tried with scripting and it's difference then i'm used to since it's of course more related to softimage.
How would you imagine the custom ppg to look like?
Who is online
Users browsing this forum: No registered users and 20 guests