Topic: Client-server data stream interception and SQL wrappers  (Read 2155 times)

0 Members and 1 Guest are viewing this topic.

Offline Mu

  • Drunk
  • Lt. Junior Grade
  • *
  • Posts: 7
  • Gender: Male
Client-server data stream interception and SQL wrappers
« on: July 15, 2004, 09:56:25 am »
So has anyone figured out how the client packets are sending instructions to the flatfile whenever a request is made?  One thing I was playing with in NWN before I stopped being interested altogether was a nifty MySQL wrapper that enveloped the server executable, sniffing string arguments that contained SQL terminology, sending the statements to the DB and, if necessary, forcing the results of a query into a place in memory that NWN could address and use.  This was a brilliant workaround for NWN's lack of persistent I/O, and way better than the halfassed solution they came up with later (this was the case for about all of their attempts to implement player-developed solutions that worked).

So if such a wrapper was capable of linking an instance of a DV2 server and a MySQL database, with another layer of scripting functionality built in (I don't know how well you can script server-level events in DV2), you can actually implement things like ridiculous OoB rules that are a pain to follow, special ship production limitations, hell even on-the-fly strategic goals that make DV2 seem more like, Idunno, a military campaign or something.

No this would not get rid of the flat file hell, but it would be a step on the way to it.  The next step would obviously be cracking the server kit and implementing direct SQL layers and eliminating the flatfile, but you know all that.  The wrapper would be a neat way for people to come up with horrible modifications nobody will enjoy while learning some SQL syntax in preparation for an evenual move to SQL.

Ex:  You have a limit in ProductionCapacity determined by an equation based on the empire's overall economy, tracked over a time period.  A shipyard turn comes up, and a DN is in queue.  The potential shiplist is queried for approval, and the applet layer detects the presence of a DN.  if((CurrentPCInUse >= ProductionCapacity) && (ShiplistEntry(X) == [one of a list of capitol ship classes]) { InterruptBuild(args); }.

Ex:  You actually want to have a research and production model for an empire based on player choices and special mission objectives, instead of a dunderheaded list of ship choices with the year they come out so everyone knows what to expect every day.  You have a humongous shiplist with production dates of 2100 for all classes, and turn off the shipyard, running your own shipyard applet instead.  Every stardate (or whenever), each empire winds up with a number of allocatable research points based on expended economy, specialized missions to go get the super duper fake sciencey Star Trek artifact, possibly influenced by individual players spending prestige on "research direction." etc.  Check to see if a new model has come out, based on the direction of reaearch, and exactly what that is.  The problems of balancing a random technology build order are worth aq treatise in themselves, but it's a nice intermediary step between what we have now in gaming (predetermined technology trees) and a realistic research path.

The real benefit to an intermediary SQL wrapper step would be the ability enforce retarded rules that I really hate having to read and remember every time I play, and that so many players (in an open server, which is what we need to keep a community alive) just never bother to read.  A system is better than a rule, since you can't argue about it; if you disengaged, the DB will time out your ability to take missions in that hex.  This I think more than anything is what drives people away from a more structured DV2... the necessity of comprehending and following a bunch of rules designed to make things fair, as opposed to fighting more.  Three Towns ran into this all the time, and where most servers would make a rule and force a DM to be a cop, we would just write a new system that took care of the behavioral issue for us.  Ask anyone who played in the Heavy Gear TKZ campaign... it's no fun doing 2 hours of paperwork, debating drop conditions for another 2 hours, and setting up matches, only to have your entire team killed by me in 4 minutes.  :P

So who all is carrying the SQL torch now, if anyone?  Because the game is already pretty much dead as far as those outside this inner circle old timer's club are concerned, and I can't see anything but the implementation of a working, stable strategic operations layer bringing them back.

Offline FPF-DieHard

  • DDO Junkie
  • Captain
  • *
  • Posts: 9461
Re: Client-server data stream interception and SQL wrappers
« Reply #1 on: July 15, 2004, 10:13:13 am »

So who all is carrying the SQL torch now, if anyone? 

I nominate you  :lol:

Bonk, Ke, Catrn, FS, and other have done a lot of work.  We have the means to fully control shipyards and all that fun stuff.   We DO NOT have a stable server kit.

I ran the last massive SQl test on the GW2 test server.  We had everyone log in for an hour (this is about as long as you can get 40 people to do something).   When 30 of us hit the same hex at the same time, my server got hit with such a nasty memery leak that I couldn't even shut it down without powering off the power supply.

SQL develpoment is pretty much on hold until the source code for the server kit is available.
Who'd thunk that Star-castling was the root of all evil . . .


Offline Bonk

  • Commodore
  • *
  • Posts: 13298
  • You don't have to live like a refugee.
Re: Client-server data stream interception and SQL wrappers
« Reply #2 on: July 15, 2004, 11:01:03 am »
Sounds sorta like DarkElf's intermediate mission reporting server for the SQL db, I remember running a few missions on it that affected the economy of a hex.

I'm not 100% we'll need the source to stabilise the SQL serverkit, I have a feeling that the skip-external-locking default of most MySQL installations has been our problem all along... though the source would be nice for so many reasons...

(hurried response at lunch here, I'll take a closer look at your post later today...)

Offline FPF-DieHard

  • DDO Junkie
  • Captain
  • *
  • Posts: 9461
Re: Client-server data stream interception and SQL wrappers
« Reply #3 on: July 15, 2004, 11:44:34 am »
Sounds sorta like DarkElf's intermediate mission reporting server for the SQL db, I remember running a few missions on it that affected the economy of a hex.

I'm not 100% we'll need the source to stabilise the SQL serverkit, I have a feeling that the skip-external-locking default of most MySQL installations has been our problem all along... though the source would be nice for so many reasons...

(hurried response at lunch here, I'll take a closer look at your post later today...)

Okay, we gotta organize a test of this if you think this is correct.   Perhaps do the same as we did on for GW3 a few days before it goes live? 
Who'd thunk that Star-castling was the root of all evil . . .


Offline Gook

  • Catbert
  • Lt. Junior Grade
  • *
  • Posts: 405
  • Gender: Male
Re: Client-server data stream interception and SQL wrappers
« Reply #4 on: July 15, 2004, 01:25:58 pm »
Jeez Mu I have no idea what that lot meant, but I note it has kept the techies here real quiet so it must be good :)

So did ya find your copy of OP yet ;)

KAT-Gook, OBS,OoW,MTA,SoK.
KAT-Fleet
Kzinti Hegemony

The God of War hates those who hesitate
.....Eurypides



Offline FPF-DieHard

  • DDO Junkie
  • Captain
  • *
  • Posts: 9461
Re: Client-server data stream interception and SQL wrappers
« Reply #5 on: July 15, 2004, 01:50:33 pm »
Jeez Mu I have no idea what that lot meant, but I note it has kept the techies here real quiet so it must be good :)

So did ya find your copy of OP yet ;)



Quick!!!  Mark this day on your calendar!!  Gook is at a loss for words!!   :rofl:
Who'd thunk that Star-castling was the root of all evil . . .


Offline RazalYllib

  • Imperial Romulan Information Service-senior advisor
  • Lt.
  • *
  • Posts: 784
  • Gender: Male
    • IRIS
Re: Client-server data stream interception and SQL wrappers
« Reply #6 on: July 15, 2004, 02:05:56 pm »
I make regular burnt offerings of cattle, chickens, small furry animals, grapes, grain, and even produce from my own garden in devout sacrifice to the Gods of Fate to bring forth the divine source code to the OP server Kit to the faithful...
Comes a time when the blind man takes your hand
Says "don't you see?"
Gotta make it somehow
On the dreams you still believe
Don't give it up
You got an empty cup
Only love can fill
Only love can fill

Offline Bonk

  • Commodore
  • *
  • Posts: 13298
  • You don't have to live like a refugee.
Re: Client-server data stream interception and SQL wrappers
« Reply #7 on: July 15, 2004, 03:47:57 pm »
Quote
So has anyone figured out how the client packets are sending instructions to the flatfile whenever a request is made?  One thing I was playing with in NWN before I stopped being interested altogether was a nifty MySQL wrapper that enveloped the server executable, sniffing string arguments that contained SQL terminology, sending the statements to the DB and, if necessary, forcing the results of a query into a place in memory that NWN could address and use.  This was a brilliant workaround for NWN's lack of persistent I/O, and way better than the halfassed solution they came up with later (this was the case for about all of their attempts to implement player-developed solutions that worked).


The client does not talk to the flatfile, only the server and it relays the information. (same for MySQL)

Quote
So if such a wrapper was capable of linking an instance of a DV2 server and a MySQL database, with another layer of scripting functionality built in (I don't know how well you can script server-level events in DV2), you can actually implement things like ridiculous OoB rules that are a pain to follow, special ship production limitations, hell even on-the-fly strategic goals that make DV2 seem more like, Idunno, a military campaign or something.

No this would not get rid of the flat file hell, but it would be a step on the way to it.  The next step would obviously be cracking the server kit and implementing direct SQL layers and eliminating the flatfile, but you know all that.  The wrapper would be a neat way for people to come up with horrible modifications nobody will enjoy while learning some SQL syntax in preparation for an evenual move to SQL.



I decoded the blob data in the ship table to enable creation of ships directly from the shiplist to populate the shipyards as desired (running on MySQL). I just need to implement and test it. (disable the internal shipyard and fill the db with the desired ships with the script of your chice.
The shipcachevector in the sercvcharacter table in the db needs to be fully decoded (still a mystery) in order to run a script to police/enforce CnC of fleets, easy to to once this blob is decoded fully. (same info present in the flatfile)

The problem with trying to talk to the flatfile directly is that it should be locked by the serverkit, I propose locating (disassembling) the serverkit to determine the memory ranges used and edit the srverkits data in memory before it commits it to the flatfile on disk, problem is these memory locations will vary from machine to machine and run to run... how to locate them reliably? I think it would be more productive to continue the efforts to stabilise the SQL kit, preferably with the source in hand...

Quote
Ex:  You have a limit in ProductionCapacity determined by an equation based on the empire's overall economy, tracked over a time period.  A shipyard turn comes up, and a DN is in queue.  The potential shiplist is queried for approval, and the applet layer detects the presence of a DN.  if((CurrentPCInUse >= ProductionCapacity) && (ShiplistEntry(X) == [one of a list of capitol ship classes]) { InterruptBuild(args); }.


Ex:  You actually want to have a research and production model for an empire based on player choices and special mission objectives, instead of a dunderheaded list of ship choices with the year they come out so everyone knows what to expect every day.  You have a humongous shiplist with production dates of 2100 for all classes, and turn off the shipyard, running your own shipyard applet instead.  Every stardate (or whenever), each empire winds up with a number of allocatable research points based on expended economy, specialized missions to go get the super duper fake sciencey Star Trek artifact, possibly influenced by individual players spending prestige on "research direction." etc.  Check to see if a new model has come out, based on the direction of reaearch, and exactly what that is.  The problems of balancing a random technology build order are worth aq treatise in themselves, but it's a nice intermediary step between what we have now in gaming (predetermined technology trees) and a realistic research path.


Almost there with the SQL kit, the actual shipyard script will be trivial once the kit is stabilised (decoding the stores blobs was hell...) THe tough part was figuring out how to create a working ship in the db with only the ionfo in the shiplist... done.. (needs further testing and possible adjustments).

Quote
The real benefit to an intermediary SQL wrapper step would be the ability enforce retarded rules that I really hate having to read and remember every time I play, and that so many players (in an open server, which is what we need to keep a community alive) just never bother to read.  A system is better than a rule, since you can't argue about it; if you disengaged, the DB will time out your ability to take missions in that hex.  This I think more than anything is what drives people away from a more structured DV2... the necessity of comprehending and following a bunch of rules designed to make things fair, as opposed to fighting more.  Three Towns ran into this all the time, and where most servers would make a rule and force a DM to be a cop, we would just write a new system that took care of the behavioral issue for us.  Ask anyone who played in the Heavy Gear TKZ campaign... it's no fun doing 2 hours of paperwork, debating drop conditions for another 2 hours, and setting up matches, only to have your entire team killed by me in 4 minutes.


If you can figure out how to modify a live flatfile without corrupting it that would be a good idea, but I think it is a greater challenge than just stabiilising the SQL kit.

Quote
So who all is carrying the SQL torch now, if anyone?  Because the game is already pretty much dead as far as those outside this inner circle old timer's club are concerned, and I can't see anything but the implementation of a working, stable strategic operations layer bringing them back.


Castrin ran a stable SQL server but no one came to stress it (I am currently running two stable servers on SQL, again no people to load it... kits are connected to an SQL server on a QNX box - but unfortunately Excel corupted the samba installation while opening a large text file over a samba share and I need to reinstall the works over the next few days so I can get samba going again to make working on the webmap scripts easier)
Where was I? Oh yeah SQL torch carriers:
Castrin, DarkElf, Kel, DieHard and myself.

Have you got permission for the D2 admin forum?
Server Admins Group--Request Access here

I would also Quiz DarkElf on the details of his SQL mission reporting server and how exactly he implemeted it. I believe he lost the source for it in a system crash some time ago... but being a smart, kind and dedicated fellow ( :-*) he might be able to reproduce the application relatively quickly.

Offline Bonk

  • Commodore
  • *
  • Posts: 13298
  • You don't have to live like a refugee.
Re: Client-server data stream interception and SQL wrappers
« Reply #8 on: July 15, 2004, 03:53:40 pm »
Sounds sorta like DarkElf's intermediate mission reporting server for the SQL db, I remember running a few missions on it that affected the economy of a hex.

I'm not 100% we'll need the source to stabilise the SQL serverkit, I have a feeling that the skip-external-locking default of most MySQL installations has been our problem all along... though the source would be nice for so many reasons...

(hurried response at lunch here, I'll take a closer look at your post later today...)

Okay, we gotta organize a test of this if you think this is correct.   Perhaps do the same as we did on for GW3 a few days before it goes live? 

I have a hunch that this has been the problem, but it is only a hunch; though a very logical one. First you need to find a platform that will support external locking and confirm that it is working, then we can try and test this... I am about to test this on my servers once I complete my QNX reinstall.

The other thing to consider is that the external locking code in the serverkit has always appeared unused, perhaps for the reason I suggest, but MySQL versions are a factor here, the support for external locking has been modified over the versions and the SQL code for the serverkit was originally written for MySQL 3.23 and EAW and was hastily ported over to OP and MySQL 4.0.17 (Thanks again Mr Ferrel!  ;D)

Definitely we should test this out under a good playerload ASAP, we may be disappointed but it certainly is worth a try.

Offline Gook

  • Catbert
  • Lt. Junior Grade
  • *
  • Posts: 405
  • Gender: Male
Re: Client-server data stream interception and SQL wrappers
« Reply #9 on: July 15, 2004, 04:08:49 pm »
Damn Bonk that was quick!!!

 :notworthy: :notworthy: :notworthy:
KAT-Gook, OBS,OoW,MTA,SoK.
KAT-Fleet
Kzinti Hegemony

The God of War hates those who hesitate
.....Eurypides



Offline FPF-DieHard

  • DDO Junkie
  • Captain
  • *
  • Posts: 9461
Re: Client-server data stream interception and SQL wrappers
« Reply #10 on: July 15, 2004, 04:19:30 pm »


Definitely we should test this out under a good playerload ASAP, we may be disappointed but it certainly is worth a try.

When we test GW3, I'll give you remote access and let you configure th MySql on the server.   

I'm sure we'll be able to talk people into doing a 1 hour test like we did last time.
Who'd thunk that Star-castling was the root of all evil . . .


Offline Bonk

  • Commodore
  • *
  • Posts: 13298
  • You don't have to live like a refugee.
Re: Client-server data stream interception and SQL wrappers
« Reply #11 on: July 15, 2004, 05:13:00 pm »
Quote
Damn Bonk that was quick!!!

I saw this post at lunch and left a quick note, I had all afternoon to think about it. (even though I didn't, I was rather busy at work, but came right back to this post as I was intrigued... I still haven't opened today's snail-mail...lol) I nuked some substandard dinner and got right to it when I got home.


Quote
When we test GW3, I'll give you remote access and let you configure th MySql on the server.   

I'm sure we'll be able to talk people into doing a 1 hour test like we did last time.

I'll give it a whirl, you're running the MySQL server on windows? (I'll investigate to see if external locking is supported on windows... I think it is... but both DarkElf and I have observed differences in the ability to control the nextID on windows vs Linux, though I don't want to use Linux as external locking is stated to be disabled on that platform, I'm sticking with QNX for now but may try FreeBSD later - the only advandage to Linux is the quality of the threads library but I think that is not a concern for the OP server, though it is for the super connection horny SFC3 SQL server...)

P.S.: not sure a Remote Desktop Connection (Windows terminal services) will do unless the account has administrative privelges, otherwise www.tightVNC.com is the way to go.
« Last Edit: July 15, 2004, 05:31:13 pm by Bonk »