DB optimization and data size reduction (unique TE feature)

[texpert]

Hello,

if you are installing Trade Expert on an already overloaded server or if you have several very huge sites on one server, you can learn here how to
optimize TE performance. If your server is not overloaded at all, you don't need to waste your time with optimizations at this point. You can optimize DB at anytime in the future when you'll need it. You won't find these database optimization functions anywhere else, they are unique to TE. We've spent months on making database optimizations possible. The end result is that you can optimize TE performance for up to 10 times or more just by swiching few parameters. And we are not missleading you when we say 10 times, it is real. If your server ever gets overloaded and you are running any other trade script, you will have to consider buying a bigger (or additional) server.

Our Trade Expert trade script uses crons as well and i can tell you that it can be optimized much further than anything else on the market. We've spent months on developing a system that allows you to adjust the whole database to your own requirements. Trade scripts nowdays consume most resources on reading large amount of data from the HDD (every X mins). Let's take a look at the example how you can optimize TE to read less data and less frequenetly from the HDD. Example: You have 10 huge sites on your web server and you're starting to experience server load problems, you can shrink the whole database for factor 2 to 60. E.g. you can set "process database" parameter to every 10 minutes, which will shrink the whole database for 10 times. Not only that database will be processed every 10 mins, it will have to read 10x less data from the HDD and that is a HUGE advantage.

Data is now processed every 10 mins, which is great but not enough. With a cron balance parameter we can simply set TE to process data on each of those 10 huge sites at a different minute (sitea.com: xx:00, siteb.com xx:01, sitec.com xx:02 and so on). This way we won't overload our server at alll. Additionally we can set cron delays for each site. This way cron doesn't need to get executed right at the beginning of a minute when also crons from some other scripts are being processed. There are several more parameters helping you reduce load on your server drastically.

You must not forget about logs as well. On a huge site 1440 minutes (1 day - default) of referrer and cheat logs can accumulate to e.g. 150 MB. Reading 150 MB on 10 scripts each minute will cause server load issues. In TE you can set for how long (in minutes) you would like to keep referrer logs, country logs, cheat logs, ip tracker logs, spider logs, ping logs etc. On an already overloaded server it makes sense to reduce log size. You can simply set TE to keep referrer logs (and other logs) for only e.g. 720 minutes. It is also possible to fully disable some of the logs which you might not need by setting minutes to 0.

regards,
Jim

[pornupload]

hi Jim
where can i find this information?

[texpert]

This post is from 2009 and it's TE 2.x related. If you are using TE3 ,you no longer need those optimizations ,because we have used a totally different approach in solving those issues. In case you're having an already overloaded server and u would like to add more TE3s to it, i have some good news for you. In the next few days ,TE 3.2 will be released officially. We have further optimized all critical modules and i'm happy to say, we've managed to reduce load for 20-30% on average server (i'm talking about a server where only TE3 is running).

When 3.2 is released, u should update.

best regards,
Jim