Was an xsibase database ever released?

[dewf]

So I'm a little late to the Softimage party, by a decade or two In my google searches I periodically come across people talking about the good/bad old days. On the Autodesk mailing list, somebody alluded to the xsibase database being publicly released at some point. Is that true? Is there any way I could access/download it? (I'm familiar with archive.org, but that's not what I'm talking about)

Put simply, I'd like to extract the data and put online a read-only archive of it for google to index. Or has this not already been done precisely because they never released a database dump? (I'm a seasoned programmer who could do this fairly easily if I had the raw data)

Sigh. As a hardcore preservationist, it annoys me greatly when stuff simply disappears Just today I was trying to read up on the tek2shoot shaders -- naturally the website no longer exists.

edit: I just discovered/read through an old thread on here, where people were making available their own personal archives. I'm currently downloading and will look through them to see if they're worth extracting data from (I happen to be really good at scraping webpages). I also emailed raffael3d directly to see if he'd be willing to give up the goods -- hopefully not too naive an approach (was there any bad blood or some reason why a better effort wasn't made to preserve the site? I don't want to upset the guy if he's the only one with that data)

[rray]

I believe there is no complete archive... xsibase disappeared without warning overnight (Well in hindsight the existing malware should've been warning enough)
Found some archives, but lots of pages (90%+) are missing in thoses archives

[Hirazi Blue]

Although I'd like to have seen it play out otherwise, I sort of understand why the whole database wasn't released. A forum database contains information that shouldn't be made public (member records for one) and I imagine, - but do not know, I hasten to add - stripping this from the forum database while still maintaining something usable to distribute might have been considered too much work...

[rray]

There's crawler software for forums that don't have a lot of custom stuff added, like this one. XSIbase was very customized so this might be why those didn't work well for it.

[mattmos]

I did ask Raffael if he would consider sharing the database a while back, but nothing ever came of it, maybe all he needs is a gentle reminder from someone with some suitable skills to recover/reformat the data. Good luck!

[dewf]

stripping this from the forum database while still maintaining something usable to distribute might have been considered too much work...

For somebody database-savvy, it's actually only a few minutes' work (dropping some tables, nullifying some fields) -- but since these were apparently artists with demanding day jobs, who were putting a lot of effort into a hobby project that didn't pay for itself ... yeah, I can see how it wouldn't have been too enticing to put any more time into it. It's also quite a free "gift" to give somebody -- years' and years' worth of data and maintenance, plus a top google position if the domain came along with it?

I'll give Raffael a few more days to respond before I try pestering some other names I've seen associated with xsibase. Until then I'll see what I can extract from these user-made .zip archives. They're definitely scant, but better than nothing!

[dewf]

So far, so good! I've extracted the metadata and raw post content from the .zip archives provided in this thread (thanks, Bullit!).

There are a total of 11,753 posts in 1750 threads (from 1,555 users). Presumably that's just a fraction of the original forum, but it's better than nothing. Some of the posts go all the way back to 2002, so that's nice.

Now the somewhat-harder work begins on normalizing/reformatting the actual post content. Basically this involves stripping HTML tags, detecting stuff like quote boxes, moving referenced image files to a centralized location, etc. A fair bit of work, but nothing that strikes me as impossibly difficult.

Fun fact: not counting the embedded images, the raw reconstructed database is only 2 megabytes when compressed. It'll be even less once I strip out superfluous HTML tags -- a far cry from the 580 megs of .zip archives I downloaded to get this information

[Hirazi Blue]

Fun fact: not counting the embedded images, the raw reconstructed database is only 2 megabytes when compressed. It'll be even less once I strip out superfluous HTML tags -- a far cry from the 580 megs of .zip archives I downloaded to get this information

The expression with the "gift horse" comes to mind... ;)

PS Couldn't a service like the Wayback Downloader be a good start? On the xsibase the "legacy" content was the most useful anyway IMHO. Note: this service isn't free and I have no idea how trustworthy this is. But I guess more people might be willing to chip in.

[dewf]

PS Couldn't a service like the Wayback Downloader be a good start?

Possibly, but based on my admittedly-superficial investigations into archive.org's copies -- the coverage isn't very good, at least for any given "capture" (crawl instance) -- lots of threads/pages missing. But of course the question is, is it a sufficient improvement upon what I have now to warrant the entirely reasonable fee? If so then I wouldn't mind paying for it myself. I could also try my hand at spidering the pages myself, since that would allow me to cover all the available captures over the years (vs. having to figure out which particular year/month/day has the best copy).

Something to keep in mind, for sure!

[Bullit]

Note that the database without images is in many threads not useful.

[dewf]

Oh, I didn't mean to imply that I was removing them for good -- I was just curious how small the actual text of the posts would be compared to the full HTML of the pages I extracted them from. Rest assured that when I'm done, all the images you saved for posterity will be included in the threads Just this morning I finished the code to generate thumbnails from the really large ones...

[rray]

Keep it up, xsibase's total post count in the end was ca. 300000 so it's a really small portion, but if it's searchable by keywords it will be very useful nonetheless.

[luceric]

Frankly I think all of this stuff is better left buried. There are all sort of work around and plug-ins that are obsolete or causing more trouble than help. And lots of misguided advice. Just ask your question in this forum and get a fresh reply.

[dewf]

Well, that's probably true -- and maybe it's a good idea not to pollute the Internet by making the archive directly indexable by search engines. (Though it will be searchable wherever it's hosted)

But the reason I'm personally doing this is not for esoteric SI knowledge, but rather simply for nostalgia and preservation. (I have a running Indigo2 sitting about a foot away from me, that I'm going to use in the near future to record high-res screencasts of old 3D software -- if that tells you anything about me)

Basically I'm fascinated with old 3D stuff, and this just seems like a fun exercise of my data-extraction abilities.

[Hirazi Blue]

I for one applaud the effort. Only living in the now seems a problematic attitude and you might be surprised at the amount of useful stuff you could still extract from that site,...
;)

[dewf]

I've been busy with some other stuff so haven't spent much time on this since my initial push. I'm trying to figure out the best way to generate a static archive of everything -- maybe just some oldschool HTML frames (one with categories, one with thread titles, another for all the posts in a thread). A single .chm file is another possibility, but I looked through the docs a bit and it looked complicated (for somebody too lazy to spend the time reading how to do it -- ie, me). Easier to just spew a bunch of individual .html files.

Here's the table of contents of what I was able to recover: http://pastebin.com/yEmbeWPQ