Handling of massive amounts of objects

General discussion about 3D DCC and other topics
Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 23 Sep 2013, 22:41

I did a little test recently. I exported a large number of objects from a max scene to obj to see where else I can load it. In max about 60% where instances, which where resolved to unique objects in obj. 2.3 gb filesize, 15 000 000 polygons, about 15000 objects.
Maya didn't recognized the obj file at all. (Yes I loaded the obj plugin, which is off by default)
Max just canceled the import without any comment at about 30%.
Softimage took 20 min to gather scene file statistics, the started to load until 32 gig ram where full and began swapping happily. I canceled the import after an hour.
Cinema 4d, the same as SI.
Blender loaded the scene in about half an hour and became unusable slow.
Modo to my surprise loaded the scene completely in about 20 min and it was completely editable. I rotated all 150000 objects 90 degree with one single parent in 30 sec. I move an object out of the hierarchie in 10 sec. I was able to orbit the scene, select single objects and focus on them. It took 30 min to select all objects.

I do not know what the reasons are for all AD products to fail this test, but it surprised me quite a bit. I at least expected maya to recognize a obj file. I expected max to be able to reimport what it exported. And most of all I expected SI to be memory efficient.
I'm now part of an endangered species...

Bullit
Moderator
Posts: 2621
Joined: 24 May 2012, 09:44

Re: Handling of massive amounts of objects

Post by Bullit » 23 Sep 2013, 23:34

Softimage has never been good with thousands of objects.

luceric
Posts: 1251
Joined: 22 Jun 2009, 00:08

Re: Handling of massive amounts of objects

Post by luceric » 24 Sep 2013, 01:56

ha yes, the daily Blender promo! But seriously.. I wonder if you would have had more chance in 3dsmax or Maya if the file was under 2 bit in size (about 2.1 gig) It's very typical on Windows where both "long" and "integers" and C library functions file pointer are 32-bit, to fail with text files around that.
Lord Futzi Voldemort wrote: And most of all I expected SI to be memory efficient.
ha ha ha ha ha ha. funny man.
Lord Futzi Voldemort wrote:Modo to my surprise loaded the scene completely in about 20 min and it was completely editable.[...]. It took 30 min to select all objects.
go modo!

User avatar
origin
Posts: 619
Joined: 09 Jun 2009, 11:59
Location: warsaw

Re: Handling of massive amounts of objects

Post by origin » 24 Sep 2013, 09:52

OPs question remains unanswered ;) or is it...
-FABRIC ENGINE? delayed data load?
-maya xgen???

User avatar
Hirazi Blue
Administrator
Posts: 5107
Joined: 04 Jun 2009, 12:15

Re: Handling of massive amounts of objects

Post by Hirazi Blue » 24 Sep 2013, 13:02

@luceric - Really, "the daily Blender promo"?
Stay safe, sane & healthy!

Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Re: Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 24 Sep 2013, 13:38

luceric wrote:ha yes, the daily Blender promo! But seriously.. I wonder if you would have had more chance in 3dsmax or Maya if the file was under 2 bit in size (about 2.1 gig) It's very typical on Windows where both "long" and "integers" and C library functions file pointer are 32-bit, to fail with text files around that.
Lord Futzi Voldemort wrote: And most of all I expected SI to be memory efficient.
ha ha ha ha ha ha. funny man.
Lord Futzi Voldemort wrote:Modo to my surprise loaded the scene completely in about 20 min and it was completely editable.[...]. It took 30 min to select all objects.
go modo!
No comment on maya's failure?
I'm now part of an endangered species...

luceric
Posts: 1251
Joined: 22 Jun 2009, 00:08

Re: Handling of massive amounts of objects

Post by luceric » 24 Sep 2013, 20:39

I did, you'd have to check if the file is recognized when it's a little under 2 gig of size. I'm not sure what "not recongnized" means, I can only guess there was an error loading

Nox
Posts: 108
Joined: 16 Apr 2013, 13:28

Re: Handling of massive amounts of objects

Post by Nox » 25 Sep 2013, 10:39

Yes, latest modo can be quite fast at rotating camera, but modeling tools are still slower than XSI/Maya/Max etc.

P.S. About Maya, 2013 version fails at importing files with filenames starting with number. For me at least.

User avatar
Nizar
Posts: 725
Joined: 30 May 2010, 22:54

Re: Handling of massive amounts of objects

Post by Nizar » 25 Sep 2013, 11:36

softiamge 2014 vewport is less reactive and robust (on my system)

Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Re: Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 25 Sep 2013, 13:09

luceric wrote:I did, you'd have to check if the file is recognized when it's a little under 2 gig of size. I'm not sure what "not recongnized" means, I can only guess there was an error loading
Not recognized is what maya said. All other apps recognized the file, so there is nothing wrong with it. And you may not know it, but this happens in maya sometimes with considerably smaller files too.

But to make it clear. I only posted this because I found this result surprising. If there can be a conclusion drawn at all, then maybe that it looks like as if tools with newer code seem to be more prepared for the future than others.
I couldn't test with houdini, so the image isn't complete. I have some other very large data sets somewhere, maybe I'll test again soon.

Oh, and yes I intend to go modo. Not now but in a in one or two years.
I'm now part of an endangered species...

User avatar
rray
Moderator
Posts: 1774
Joined: 26 Sep 2009, 15:51
Location: Bonn, Germany
Contact:

Re: Handling of massive amounts of objects

Post by rray » 25 Sep 2013, 15:54

I used to have more luck importing big obj files using the fbx importer (menu "file->import fbx/switch to obj file format"). A bit obscured :)

Also with such files it's sometimes a good idea to have no 3D viewport open before importing if you plan to do poly redcution or something.
softimage resources section updated Jan 5th 2024

Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Re: Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 26 Sep 2013, 12:03

OK, somehow maya managed to read the file on another machine after all and now I get this:

// Error: line 0: OBJ file line 14635518: index out of range for face creation. //
// Error: line 0: Error reading file. //

Again all other apps read the file (max to a certain point). I think I could eventually load it in SI if I upgraded to 64 gig of RAM.
I'm now part of an endangered species...

luceric
Posts: 1251
Joined: 22 Jun 2009, 00:08

Re: Handling of massive amounts of objects

Post by luceric » 26 Sep 2013, 17:35

Lord Futzi Voldemort wrote:
luceric wrote:I did, you'd have to check if the file is recognized when it's a little under 2 gig of size. I'm not sure what "not recongnized" means, I can only guess there was an error loading
Not recognized is what maya said. All other apps recognized the file, so there is nothing wrong with it. And you may not know it, but this happens in maya sometimes with considerably smaller files too.

But to make it clear. I only posted this because I found this result surprising. If there can be a conclusion drawn at all, then maybe that it looks like as if tools with newer code seem to be more prepared for the future than others.
I have mixed feelings. I think it's a test of the .obj importer. If Maya can't handle huge .obj files, it just means that no studios is using it that way, because otherwise it would be fixed. Autodesk is in daily meetings with dreamworks, weta, dysney, ilm, lightstorm. Scalability is always part of the discussion; there are people here dedicated to working on that. I don't use Modo (obviously) but it was said that Modo is not scalable, and the user base was hoping The Foundry will do something about it and fix it - there was a lot of discussion about that around The Foundry acquisition, and talked about in the fxguide interview by John Knoll. (Maybe they mean another kind of scalability - reference models perhaps?) It's not necessarily a question of newcode/old code. You can load directly alembic to GPU in Maya, that's all new code, as is all the new asset management and Viewport 2.0 work, and the future opensubdiv work and bifrost work. Also, the softimage code is newer than maya and (much) newer than max, but struggles with larger number of objects, possibly because we focused on structures to support operator stacks, property inheritances (which is used visibility, materials, etc), and other architectural goodies, which are not important when you want to load a 15,000 static props.
Lord Futzi Voldemort wrote:I couldn't test with houdini, so the image isn't complete. I have some other very large data sets somewhere, maybe I'll test again soon.
Oh, and yes I intend to go modo. Not now but in a in one or two years.
I couldn't wrap my head around Modo modeling workflows when I tried it, for some reasons, but many people can, for sure.

Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Re: Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 26 Sep 2013, 18:44

I was just able to load the data in maya as fbx (which is good when your client has the data in fbx format) and I have to admit, maya was quite responsive and the viewport lightning fast until it froze completely.
I'm now part of an endangered species...

Chris_TC
Posts: 411
Joined: 22 Mar 2010, 17:43

Re: Handling of massive amounts of objects

Post by Chris_TC » 26 Sep 2013, 20:15

luceric wrote:Also, the softimage code is newer than maya and (much) newer than max, but struggles with larger number of objects, possibly because we focused on structures to support operator stacks, property inheritances (which is used visibility, materials, etc), and other architectural goodies, which are not important when you want to load a 15,000 static props.
The performance problems with 10,000+ objects have bitten me a handful of times. But I really wouldn't want to miss out on the things you mentioned above. These are things that help me big time in my day-to-day workflow, so I can live with having to work around the many-objects-issue every now and then.

Lord Futzi Voldemort
Posts: 440
Joined: 06 Jun 2009, 14:01
Contact:

Re: Handling of massive amounts of objects

Post by Lord Futzi Voldemort » 27 Sep 2013, 13:24

I managed to load the data in softimage via fbx as well, which seems much more memory friendly than the hopelessly outdated obj importer.
Performance is surprisingly not much worse than maya, except from the viewport. Viewport 2.0 in maya is great but also unreliable in my experience.
I played half an hour with the scene with no issues other than slow screen updates when selecting which were still in a tolerable range given the huge amount of data. After I saved the scene it reloaded pretty fast.
So after all I think it looks not that bad after all for SI. Max however becomes rather unstable and laggy even when only selecting single objects or switching editors, it freezes a lot and I have to restart it after using a heavy scene before I load something else to prevent it from locking completely.
Maya has it's own issues. I wasn't really surprised that it couldn't read the obj file, after all it sometimes refuses to read simple jpgs or tifs often enough. It could be a coincidence that it locked up with this data, but it did, and such coincidences are not seldom. For Si however the bottleneck clearly is the poor viewport. I wasn't able to get any lighting like headlight for the shaded view until I reloaded the scene. Also the importers need heavy work. (I wonder when that will happen ;))
I was surprised blender could load the data at all although it became unusable slow. C4D can handle large amounts of data. I had scenes with a similar amount of objects which it handled fairly good. I think the obj importer was also the problem here (didn't test fbx).
The really big surprise was modo, because given the complaints of modo users regarding performance I didn't expect it to perform in a tolerable range at all. And it was one of the two apps that had no problems reading the obj file. It is definitely worth to keep an eye on it, although I don't think the foundry intends to compete against max maya or si with it. We will rather see it as a part of the foundry's full production framework consisting of nuke, catana, mari and modo, all interlinked directly or via hiero. (something that can bring AD in serious trouble IMHO)
Attachments
15307.jpg
I'm now part of an endangered species...

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests