Dismiss Notice

Welcome To CK5!

Registering is free and easy! Hope to see you on the forums soon.

Score a FREE t-shirt and membership sticker when you sign up for a Premium Membership and choose the recurring plan.

Looking for ideas on computer server load balancing algorithm

Discussion in 'The Lounge' started by 73k5blazer, Jul 25, 2006.

  1. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    Ok,
    I've got 5 servers here. The application hitting them is extrememly memory intensive, taking ~340mb/user session connected. I wrote a load balancer for it 2 years ago based on memory only, and since there was plenty of memory, it worked flawlessly for that time, but users are growing and now machines are starting to page. They can handle it, but since they won't be getting a new server for a few weeks here, some of the servers have more processors than others, and thus is a bettery choice after the machine starts paging, but obviously only up to a point.
    So here's what we got

    server 1 - 28gb ram 4 cpu (started to page at 86 users)
    server 2 - 24gb ram 4 cpu (started to page at 75 users)
    server 3 - 16gb ram 2 cpu (started to page at 56 users)
    server 4 - 16gb ram 2 cpu (started to page at 56 users)
    server 5 - 16gb ram 2 cpu (started to page at 56 users)

    My original load balancer was just using memory, and we had plenty of room to grow, so memory was the only consideration. Now at 325 sessions, all memory is used and servers start paging, slowing the application a bit. The more cpu machines can take more load and reduce the amount of paging requried.
    Any ideas on how best to mathmatically load balance these things after they start paging?
     
  2. Avery4jc

    Avery4jc 1 ton status Premium Member GMOTM Winner

    Joined:
    Oct 24, 2005
    Posts:
    10,760
    Likes Received:
    75
    Location:
    Fresno, Ca.
    I don't know about your question but I do know that is a crap load of ram. That must have cost a small fortune!

    -Avery
     
  3. CarnivoreMan

    CarnivoreMan Registered Member

    Joined:
    May 29, 2005
    Posts:
    46
    Likes Received:
    0
    Location:
    Ferndale, WA
    So uh.... ummm...



    Obviously this isnt a copy and paste deal.. I dont know the language your writing in.. and if I did I wouldnt remember the syntax if I ever did know it in the first place. :D ... but maybe this can help get some ideas flowing.

    Of course the variables cant be reset between clients joining up, they need to be saved somehow. I dont know where your scrips reside and how you can store variables for long periods.. I only have a few semesters of coding in a few languages and I've forgotten most everything.


    and a note... I began writing this all this morning, then finished it up now.. so uh.. It may not all may sense.. and perhaps it doesnt even answer your question because I have absolutelly no idea how to manage server load with scripting..... I just play with Active Directory:dunno: and actually, I dont really manage that crap with AD either.. I just disperse the rolls at install.. Good luck man.

     
  4. readymix

    readymix 3/4 ton status

    Joined:
    May 30, 2004
    Posts:
    6,706
    Likes Received:
    26
    Location:
    Murrieta, California
    There was a time when I was not confused......

    That time has obviously passed:haha::haha:






    Good luck with whatever the hell you are talking about......
     
  5. CarnivoreMan

    CarnivoreMan Registered Member

    Joined:
    May 29, 2005
    Posts:
    46
    Likes Received:
    0
    Location:
    Ferndale, WA
    Clarification on what my script supposedly does.. Just cycles through the servers evenly adding one user to each server until each server hits its own specific max... at that point um.. well it looks like it'll just stop responding.. :D
     
  6. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    Well, thanks for the reply. I was thinking about this alot today, while I was watching all the servers paging, and what was I watching, not the CPU,but the amout of paging space left. So, really, the CPU is not very important I guess, memory (and paging space) is, as each person connects, kiss 340mb goodbye, whether is real or virtual. And due to the extremly poor programing of the "enterprise" application itself, we only see 120 given back when you disconnect. Sweet. Anyhow, for now I added a a check for free paging space and if real memory is out, it uses the lowest usage of paging space then.
    Language is simply korn shell on an AIX box (UNIX). It's called from a php web startup that goes and gets the required stats from each server.

    Paging is brutal though....my enjoyment for leaving for 18holes at 4 in the afternoon is ruined when I know those things are burning disk.....
     
  7. 1-ton

    1-ton 1/2 ton status Premium Member

    Joined:
    Apr 17, 2004
    Posts:
    2,880
    Likes Received:
    159
    Location:
    Las Vegas, Nevada
    Are you using a Microsoft Server product? Microsoft is notorius for running a lot of "Services" in the background, which takes up a lot of memory. If you are using a Microsoft Server product, one way to free up some memory is to turn off any unessesary Services. If you go to...Start>Program>Administrative tools>Services, you can turn off any unessesary Services there, but make sure you know what Services are needed and which ones are not, or you could really screw things up.
     
  8. CustomChevy

    CustomChevy 1/2 ton status

    Joined:
    Jun 29, 2005
    Posts:
    4,328
    Likes Received:
    0
    Location:
    Kitchener, Ontario, Canada
    what database is eating such ram?

    it sounds like no matter what you do to balance them you are going to be paging. Short of a new server, I'd be looking at ways to make the app more efficent.


    Also might try a little optimization server side, only half the RAM unloading when a session is closed is suspect.


    Sorry, I was never good with code, but those would be my thoughts on the matter.
     
  9. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    Nope, no windows. These are AIX servers running only this application. At boot, AIX is only taking 380mb of memory.
    It's not the database, DB2 is fine and happy on it's own server taking 12gb of ram, which is just about right for the amount of data being pushed through there.
    It's a PDM application, called LCA (LifeCycle Applications). It's Dassault's PDM. Dassualt is the author of Catia CAD software. Yeah, we know the code sucks. Dassault code has always been notorious for memory leaks, crashes, bugs, more so than the majority of software out there I would say. With this application, you start on the clients (windows), it connects to the app server via orbix, and it starts your server session there. This manages all the connections to the database, connections to vaults (where catia models are actually stored), and all the security access checking and a few other tasks. The client user just sees a pretty java window with his product structure in there and the user can manipulate his structure, add new parts, configuration filters, load models into Catia...etc.. These server side sessions take 350mb. Dassault says that's perfectly acceptable, they recommened 32-way 300gb RAM IBM server for this customer. I work for IBM, that's close to a million bucks for a server of those specs. We shouldn't need that for a miniscule 400 people.
    Oh yes, we know...we know oh so well how bad the code sucks. I've been working with Dassault API's for 10 years, half of them don't work as advertised,or crash when you attempt to use it. It's a pick your battles thing with Dassault though, there's so many battles, you need to have the critical ones fixed, this isn't critical yet, but it will be soon..

    Back to the load balancing, I know I've seen load balancers out there that let you weight RAM, CPU speed,no. of CPU's, and I've been searching and searching for what algorythms they use, I can't seem to find a decent one..
     
  10. CarnivoreMan

    CarnivoreMan Registered Member

    Joined:
    May 29, 2005
    Posts:
    46
    Likes Received:
    0
    Location:
    Ferndale, WA
    hmmm.. have you tried rebooting? That fixes a lot of my problems. :D
     
  11. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    The servers reboot every night. This gets us all our memory back from the crappy dassault application.
    Impossible to reboot during the day, we'd have to kick everyone on that server off.
     
  12. jekquistk5

    jekquistk5 Weld nekid Premium Member GMOTM Winner

    Joined:
    Jan 18, 2004
    Posts:
    11,005
    Likes Received:
    838
    Location:
    Pinckney, MI
    Hello fellow IBMer...Short of adding more ram to them ~15 gb to get ya going, its going to page. What are the specs on the HD's if you have extra hard drive space try striping an array 0+1 or raid 5 with some 15k scsi ultra 320 or a SAS setup just for the paging. I'm thinking that may be your bottle neck. The thing is the program is crap, unless you find a better program were you can tweak it a little better. DB2 is fun :)

    edit: And how much space to you have reserved for paging, is it too small? a realistic pagefile should be 1.5 times the ram leaning towards 2 times with heavy mem use. But I'm sure you allready know this
     
  13. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    There are 2 discs per machine, they are 15k ultra 320's. They aren't on a raid, the 1st disk is rootvg and is OS only & application. the second is paging, apache, db2 client, and a few other very low consuming things. They are newer 1u servers, so 2 disks inside is the physical limit, we'd have to put a tower on them to have a good raid setup.
    Currently I have 9gb ofpaging each. I'm not so sure on the recommendations for amount of paging space. 9 is plenty enough to cover what the real doesn't. I know back 13 years ago, when memory was ridculously expensive, the AIX machines I had catia on had 2 400mb drives, and 120mb RAM. Then it makes sense to have a 240mb paging space. But when the server has 28gb, you can't realistically have a 56gb paging space. Well, I guess you can, but it's too much. Out of the 9gb, only 3-5 gets used by the end of the day.
    Unless, you think simply have the wider space will make it page in and out faster?
     
  14. jekquistk5

    jekquistk5 Weld nekid Premium Member GMOTM Winner

    Joined:
    Jan 18, 2004
    Posts:
    11,005
    Likes Received:
    838
    Location:
    Pinckney, MI
    that's what I was thinking but your saying that only 3-5 gets that's kinda weird because if you have 325 concurrent connections using 340mb per connection that uses up 108GB:eek1::eek1: and your servers have 100 raw, not to mention the 12 used by db2. So that's why I was thinking if you had a small page file a lot time could be wasted purging data, that can be eliminated by having a bigger page file.

    But if you have 9gb per server you should be alright.. You should have about 19GB that are paged so it does work out to 3.8 GB so.. everything sounds right your just running into disk access and latency times that you do not have with solid state.
     
  15. 73k5blazer

    73k5blazer Unplug the matrix cable from the back of your head Premium Member

    Joined:
    Dec 28, 2001
    Posts:
    4,987
    Likes Received:
    38
    Location:
    The Rustbelt
    DB2 is on it's own server. Differnent machine. A Very happy machine. I wouldn't dare run a dassault product on the same server as my DB2 server. :)
    I'm watching the paging stat's, memory, cpu and disk realtime on the app servers. The disks don't seem to be very busy. Every few seconds they bop to 30-40%, stay there like 1-2seconds, and go back to 0. That is the paging, but it's not pegged out at 100% or anything.
     
  16. jekquistk5

    jekquistk5 Weld nekid Premium Member GMOTM Winner

    Joined:
    Jan 18, 2004
    Posts:
    11,005
    Likes Received:
    838
    Location:
    Pinckney, MI
    Well you got me stumped.. find a spare box and add it to the farm, thats all I can think of right now. I have 3 servers for my project that I'm not using too bad it can't be reassigned :(


    man if its only getting to 30-40% for a few seconds then what the heck is the slowdown??? It shouldn't be that slow.. I can go talk to the PCOMM and Citrix server guys see what they think. They allways seem to get me pointing in the right dirrection
     

Share This Page