XSI Base down?

[xeraphine]

Here is the backup (compress 123mb, real 600mb) of all the forum, its a mess, but nice to have for consulting in your hard drive.

http://www.mediafire.com/?i1vqznartel1gp9

[Falam]

Here is the backup (compress 123mb, real 600mb) of all the forum, its a mess, but nice to have for consulting in your hard drive.

http://www.mediafire.com/?i1vqznartel1gp9

Literally the entire site ? What would be nice is if it was organized and hosted online.

[xeraphine]

I think thats not possible, i download using htttrack the result its thousands of web with the name indexa024, index64b etc, download at see by yourself.

[druitre]

Thanks for uploading guys, I'll have a look.
Maybe it's good for creating one of those 'useful tips & tricks' sticky pages.

[Yanik]

Here is the backup (compress 123mb, real 600mb) of all the forum, its a mess, but nice to have for consulting in your hard drive.

http://www.mediafire.com/?i1vqznartel1gp9

Literally the entire site ? What would be nice is if it was organized and hosted online.

As xeraphine states it isn't possible.
What would be possible is scraping each html file and putting it in a database and later add it to the Softimage Community database.

To scrape it you need to find/match strings which are common in each topic and posts. Below I'll explain where to scrape the data from and how it should be stored in the database.

There are 3 different tables which must be filled:

Table 1: Topics
http://www.phpbbdoctor.com/doc_columns.php?id=21

topic_id mediumint(8) unsigned
auto_increment
forum_id smallint(8) unsigned
topic_title char(60)
topic_poster mediumint(8)
topic_time int(11)
topic_views mediumint(8) unsigned
topic_replies mediumint(8) unsigned
topic_status tinyint(3)
topic_vote tinyint(1)
topic_type tinyint(3)
topic_first_post_id mediumint(8) unsigned
topic_last_post_id mediumint(8) unsigned
topic_moved_id mediumint(8) unsigned

Very important: Only scrape topic info from the first page. To find out if you're on the first page you can search for:

Code: Select all

[<b>1</b>]

The forum_id field should point to a forum ID which should be added here to hold all posts. I'd put them all in 1 forum for simplicity.
The topic_title and topic_views can easily be scraped from this string, the string between {} is variable:

Code: Select all

<font size="2" class="text1" color="#FFFFFF"><b>&nbsp;Topic: {topic_title}</b> &nbsp;(Read {topic_views} times)</font>

The topic_time can be found in the first user post, see more about that in table 2 description. topic_replies, topic_first_post_id and topic_last_post_id can not be added till all posts have been scraped.

The topic_poster field can be left blank so it's an unregistered user. The topic_status, topic_vote, topic_type and topic_moved_id can be all left untouched.

Table 2: Posts
http://www.phpbbdoctor.com/doc_columns.php?id=9

post_id mediumint(8) unsigned
auto_increment
topic_id mediumint(8) unsigned
forum_id smallint(5) unsigned
poster_id mediumint(8)
post_time int(11)
poster_ip varchar(8)
post_username varchar(25)
enable_bbcode tinyint(1)
enable_html tinyint(1)
enable_smilies tinyint(1)
enable_sig tinyint(1)
post_edit_time int(11)
post_edit_count smallint(5) unsigned

The topic_id you can find by searching the already added topic and search for it's name and time(see below) by scraping:

Code: Select all

<font size="2" class="text1" color="#FFFFFF"><b>&nbsp;Topic: {topic_title}</b> &nbsp;

You can select the forum_id from that query also.

The poster_id field can be left blank so it's an anonymous user.

The post_time can be scraped by searching for:

Code: Select all

<font size="1">« <b>Reply #{reply_number} on:</b> {post_time_month} {post_time_day}, {post_time_year}, {post_time_hours}:{post_time_minutes}:{post_time_seconds} {post_time_am_pm} »</font>

If the reply # is equal to 1 you have found the first post, but if you want to be 100% sure you should sort the posts on date.

For the poster_id you should set it to 7f000001, which is equal to 127.0.0.1(localhost). You won't know the ip address so you need to fill it with a dummy value.
post_username can be scraped by matching:

Code: Select all

<acronym title="View profile of {post_username}">{post_username}</acronym>

All other fields I'd keep default. After you're done with all posts in one topic you can update the topic_first_post_id and topic_last_post_id and topic_replies.

Table 3: Post texts
http://www.phpbbdoctor.com/doc_columns.php?id=10

post_id mediumint(8) unsigned
bbcode_uid varchar(10)
post_subject varchar(60)
post_text text

The last step is to scrape all post texts. It's html so you'd need to convert it to flat text or back to bbcode. You need to scrape everything between:

Code: Select all

<hr width="100%" size="1" class="windowbg3" />

and

Code: Select all

</td></tr><tr><td class="windowbg2" bgcolor="#F8F8F8" valign="bottom">

The post_subject is not very important I think but can also scraped from:

Code: Select all

<td align="left" valign="middle"><img src="YaBBImages/xx.gif" alt="" /></td><td align="left" valign="middle"><font size="2"><b>{post_subject}</b></font><br />

I have no clue what the bbcode_uid field is for, but it is required.

To find the matching post you need to make a query and search for the same topic title and post time.

Also don't forget to strip all spaces/tabs/newlines otherwise matching might be a lot harder.

That's basically it. Anyone feel like doing it ? You'll love writing regular expressions .

Edit fixed small mistake

[Hirazi Blue]

@luceric – As the Admin tasked with manual account activation I think the main influx already has happened over the last couple of months. Ever since the XSIBase ran into severe problems at the end of last year, many have already taken the time to register an account here as well. So yes, there has been a noticeable influx, but it has been going on for quite a while now. I don’t expect too many new registrations on account of the xsibase now actually having closed. But I could easily be mistaken.

@xeraphine – Briliant and many thanks.
Combine the unzipped folder with a local search engine like docfetcher, restrict its search scope to only the forum folder, let it build an index, and you’re more or less ready to go. It’s not ideal, but the on-site search engine of the XSIBase wasn’t either… There are obviously other programs that do the same, but I just happened to have tested it with docfetcher first.

@Yanik – as you seem to be quite proficient in the wonderful world of webforums, would you be available for possible “internal” Q&A’s if the need were to arise (in case of an exploding forum for instance)?

[Yanik]

@luceric – As the Admin tasked with manual account activation I think the main influx already has happened over the last couple of months. Ever since the XSIBase ran into severe problems at the end of last year, many have already taken the time to register an account here as well. So yes, there has been a noticeable influx, but it has been going on for quite a while now. I don’t expect too many new registrations on account of the xsibase now actually having closed. But I could easily be mistaken.

@xeraphine – Briliant and many thanks.
Combine the unzipped folder with a local search engine like docfetcher, restrict its search scope to only the forum folder, let it build an index, and you’re more or less ready to go. It’s not ideal, but the on-site search engine of the XSIBase wasn’t either… There are obviously other programs that do the same, but I just happened to have tested it with docfetcher first.

@Yanik – as you seem to be quite proficient in the wonderful world of webforums, would you be available for possible “internal” Q&A’s if the need were to arise (in case of an exploding forum for instance)?

DocFetcher seems really handy. Solves the whole problem.

@HB - I'd have no problem assisting, but I'm afraid my PHP and MySQL knowledge(the tech on which this forum runs) is not that good. I don't know anything about PHPBB or any other forum system's internals, I just googled "PHPBB database structure". I'm an experienced Python/Django webdeveloper, but it can't hurt to ask.

[rray]

Could be I've messed something up during unpacking the archive, but it looks to me the rar archive only contains page 1 of most of the subforums

[xeraphine]

Could be I've messed something up during unpacking the archive, but it looks to me the rar archive only contains page 1 of most of the subforums

Is not the case, am not an expert but, try the local search engine docfetcher, like Hirazi Blue say, it works fine (thanks Hirazi).

[rray]

Hm checked again, docfetcher works great with this, but the archive only contains at most 1% of the old xsibase threads. You can check by opening index.html and going to the general forum, page 2 and try to click any thread there, same on page 3 to 240 (didn't check all ). Something must've gone wrong with httrack I'm afraid.

[xeraphine]

I know what u mean, that was my first impresion the first time i check this, but i think that the information is there, the problems is with the "level of links" or "how many clicks to the tread", dificult to explain maybe someone could explaint better an corroborate that the information its missing. By the way, i have more than the forum folder, in fact the folder of the proyect is 4gb with all the pictures and videos hosted in other sites, and the articles, jobs and news. If anyone want this i upload to.

[bottleofram]

Disappointed that not a single word of explanation was given (unless i missed it). It was an important knowledge base after all. Sure, some would have you believe Softimage started around 2008, but I've discovered this to be a complete lie! Dammit.

Anyway, there are quite a few broken links on rray.de/xsi.

[EricTRocks]

@bottleofram: have you reported it to reinhard? also which ones you looking for someone might be able to send it to you.

If people didn't see the writing on the wall a few months back with the whole site is infected with malware and being down for weeks and weeks I don't know what else could have been done. As far as I have known, Raf hasn't been maintaining the site since I've been around the Softimage community. The amount of time to maintain a site is enormous. Even more so when you get a virus / malware. Its not fun.

We still have this forum and can build it up.

[rray]

I don't think there was any announcement, bottleofram. Quite disappointing indeed. I'll remove all the xsibase links from rray.de with the next update. If someone's willing to help finding appropriate replacement links (or have some memory what was in the threads that could be interesting for the description), that would be cool! just submit them using the rray suggest form or pm.

[bottleofram]

@bottleofram: have you reported it to reinhard? also which ones you looking for someone might be able to send it to you.

Ah, I was referring to the "discussion" links, Eric. Often they would contain lots of info on proper use of the plugins and more. Thankfully, Rainhard collected all the plugin files so i believe the important bits are safe.

Hurray for rray!

[bottleofram]

I don't think there was any announcement, bottleofram. Quite disappointing indeed. I'll remove all the xsibase links from rray.de with the next update. If someone's willing to help finding appropriate replacement links, that would be cool! just submit them using the rray suggest form or pm.

There are around 63 links on rray.de referencing xsibase at the moment. If those discussions need be recovered/saved, waybackmachine probably has most of them cached. I'll have a better look over the week to see how i can help.