About Complete Resources page problems

[Kilyle]

We do have some sort of backup in case the host up and deletes the files, yes?

That sort of thing happened over on Blip.tv, where a lot of people lost their videos.  Of course, they gave warning (by a few months I guess) of the change in policy, but not enough warning to backup everything that was on there (which one group attempted to do - they got about a tenth of the total content).

So please do be careful.  I'd hate to see this site lose data.  I may be too busy to program these last few years, but I still come back here now and then and I always feel great to see this site is still up and running.  I'm hoping to get my nephews (now 10) into programming soon, and with any luck we'll be getting a Raspberry Pi soon and working on retro games within a couple years.

[Alex]

We're still fighting. And they're not proving to be much helpful to identify what the cause of this "4TB" is.
So, I am proceeding with tests, as the only thing they give me is a generic memory usage per day thing - which means every test I make needs at least 1 day just to show whant happened.

Right now I temporary put down the entire CR area. This way, by tomorrow, I should know if the amountof RAM was related to those or something else.

[Felix-0]

Man that's strange. It's not like were hosting entire videos of data. I really don't see 4 TBs. I just don't see where that number can possibly come from.

[Alex]

Well, don't make cofusion.
They're not talking about 4TB of Bandwidth, but 4TB of used memory for script running. Still high though.

However, as they're not providing something useful, I made my own checks. I found that they allow me to download the raw acees logs, so I was able to write  script of mine to parse them.
Some interesting data I found, related to Sunday.
To the sensible pages I was checking (mostly: complete resources, the generators and the forum) we had:

- 236368 hits from regular IPs
- 41142 hits from bots and crawlers

About the pages visited by regular IPs:
- images from complete resources (non php files): 32003
- resources.php (the file browsing CR pages): 2141
- resources_download.php (the file to click to download a CR): 371
- viewer (the generators file showing each of the small images we have there): 184076
- forum (total accesses): 17777

So, what now?
For now I've added a mroe strict robots.txt, let's see if that will prevent all those hits from bots.
Out of that, the only real big PHP thing I see there is the viewer. I'll have to dig to separate the IPs accessing specifically that.

And that's it for now. Phew!

[Alex]

Next step. No replies from them, but I played a little with math.
If you're curious about that, let me share some math
This is the latest reply I sent them:

Dear staff,

This reply includes some important new information.
So, please revise it fully - this time an appropriate reply on the written topics will be needed.

I made a parse of our log file relative to Friday January 31th, this file:
ex20140201000001-72.167.131.126.log
For that day, these are the data you sent me, as demonstration of our violation:

Date Username # procs CPU time Memory Usage
2014-01-31 charashosting 219836 19545 secs (0.09/proc) 4125.27 GB (19.22 MB/proc)

And here are the results and tech data you'd need to read about.
For each specific entry I've also added the consumed memory per hit detected by placing memory_get_usage(true) in the page.

File: ex20140201000001-72.167.131.126.log
Total accesses: 388395
Detailed accesses by sensible locations:
- resources: 186767 (these are not php files, but direct links to images)
- resources.php: 13669 - 256Kb (php file showing pages)
- resources_download.php: 3219 - 256Kb (php file generating images)
- viewer.php: 132394 - 256Kb (php file generating images)
- forum: 19667 - 4,25 to 5,5 MB (average 4,875, all of the forum's scripts)
- php generic: 17687 - unknown (other unmatched PHP entries)
- others: 14992 - unknown (other unmatched entries, but NOT php files)

So, picking the specific PHP files I monitored, the used REAL memory size is:
(13669+3219+132394)*256 = 38216192 KB = 37320,5 MB = 36,45 GB = 0,035 TB
19667*5,5 = 108168,5 MB = 105,63 GB = 0,1 TB

This makes a total of 168949 PHP calls for 142,08 GB registered used script memory.
In your log you say we had 219836 processes for a total of 4125,27 GB. Subtracting my results, what's left is:
219836-168949 = 50887 procs
4125,27-142,08 = 3983,19 GB = 4078786,56 MB

So, accordingly to your logs and my pure math detailed here, the unchecked pages should be:
4078786,56/50887 = 80,15 MB/Proc.
Honestly, don't you think this is a little... absurd???

Moreover, a simple phpinfo shows we have a limit of 64M per process. So 80,15 MB/Proc cannot simply be.

With these numbers, the issue seems to be on your monitor.
I am sending you this because I want to make clear why I keep saying it's not something I can fix myself.
Even if you do not provide "help on custom coded solutions", it's clear there is something not matching here.

So, if you really want the issue fixed (and you surely want), you'd really need to help me understand how this math is even possible.

Looking froward to hearing for you,

Alexander Bonivento

[Archem]

Math. Hard to argue with numbers. Hopefully, that'll get 'em to cooperate.

[Moosetroop11]

Love that reply : p

[SaiKar]

Frickin stick it to 'em, bro.

[Alex]

UPDATE. Their reply.
And I'm not kidding, I get it with this very same formatting:

Dear Sir/Madam, Thank you for your response. We have verified that your CPU and Memory Usage are still too excessive for shared hosting. At this point, we will not provide a new report on your usage, as it appears this has caused some confusion throughout our correspondence. However, your top accessed URLs are below, as this information may be helpful during your review: Top accessed unique URLs sorted by hitcount: 2137 - GET charas-project.net/charas2/viewer.php 1729 - GET charas-project.net/resources_down.php 343 - GET charas-project.net/index.php 210 - GET charas-project.net/charas2/charas_r.php?lang=en 200 - GET www.charas-project.net/forum/Themes/charas2/css/index.css?rc5 191 - POST www.charas-project.net/forum/index.php?action=register2 190 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=115 186 - GET www.charas-project.net/forum/Themes/default/scripts/script.js?rc5 186 - GET www.charas-project.net/forum/Themes/charas2/scripts/theme.js?rc5 165 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=14014 158 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=4965 157 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=8106 157 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=4888 157 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=4205 157 - GET charas-project.net/charas2/charas_r.php?checkurls=1&lang=en&new=&newtext= 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=9650 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=4524 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=3833 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=22134 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=21026 156 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=14749 155 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=9649 155 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=5852 155 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=4887 155 - GET charas-project.net/charas2/viewer.php?base_sizeset=1&row=3&scale=2&img=3119 To resolve this matter, you will need to either find a solution to ensure that your application is suited for shared hosting, or move your website to a Virtual Private Server or Dedicated Server. You can find the details of our shared hosting limitations in the Hosting Agreement at the following link: http://www.godaddy.com/agreements/showdoc.aspx?pageid=HOSTING_SA Your hosting account is scheduled for review on February 7, 2014. However, please note that if we find your account begins negatively impacting the shared hosting server, your account may need to be suspended before that date. We appreciate your cooperation and understanding in this matter. Please let us know if there is anything else we can assist you with. Thank you, Tiffany H. Professional Hosting Support As we value your feedback about the service you received, please take a moment to complete the short survey linked at the bottom of this email.

For your information, this is the reply I quickly sent them:

Tiffany,

As I asked for, replies on the specific topics I wrote were expected.
Please look at them, and try to explain the math I wrote.
I was extremely specific in my analysis, while your reply still add nothing to the discussion.

BTW, from you TOS:
In the event the bandwidth or disk space usage of your website presents a risk to the stability, performance or uptime of our servers, data storage, networking or other infrastructure, you may be required to upgrade to a VPS or Dedicated Server, or we may take action to restrict the resources your website is utilizing.
Sorry, but in there I can't really read anything about "shutting us down". Eventually "restricting our resources".
If that's the case and if we're overusing some resource, why can't you simply do as described and limit them? IE, forcing a cap?

Regards,

Alexander Bonviento

And just to be safe, I'm considering to take it back and move everything into a VPS of mine I already have (for my work, I have a full virtualization server with some space).
Right now I'm starting to move files, just in case those crazy horses will do something stupid...

[DragonBlaze]

What host are you using that gives you customer service this terrible?

[Alex]

You can see it the reply. It's the terrible GoDaddy... I know they aren't the best (at all!) however, at least till now, they did good for 6 years.
Looks like they want to lose a customer

[DragonBlaze]

Yeah I just caught that myself. I've heard other bad things about them.

Doing some very quick math myself. The top two pages have ~4000 hits, but then decline steeply from there. So I'll use 10000 hits as a rough estimate for total hits. To use 4TBs of memory, that means that each page load would use roughly 400 MBs of memory. I realize that this site uses more memory than the average site because a lot of the UI is calculated on the server, but to use 400 MBs of memory each page load is.... essentially impossible under normal circumstances.

Do you have any large recursion algorithms that may eat up a lot of memory? Otherwise is it possible that a long running script has a memory leak? Apparently PHP's memory management isn't the best out there, and you can get a memory leak with circular dependencies.

[Alex]

I thought that as well.
So I've downloaded the full access log and made a parser of it myself. Extracting every single line of every single access.
That's there the math from my prev reply came from.
However, in any case, phpinfo confirms there's a memory limit of 64MB, so something higher than that per script would not be possible in any case.

Also, it's ages no one is doing coding stuffs in here. I found most users being bots and (possibly) spammers.
But again, I've analyzed all of the possible pages from the logs.

At this point, if they seem to ignore the math I provided, am I allowed to think I'm talking with someone... stupid?

[Cerebus]

Signature related.

[DragonBlaze]

At this point, if they seem to ignore the math I provided, am I allowed to think I'm talking with someone... stupid?

It's not that you're allowed to, it's that you ought to.

I venture one of three things happened.

1.) Tiffany made a mistake, looked at the report for the whole server instead of the charas shared instance, mistakenly thought the server's stats where solely charas stats, them simply never bothered to look again. (or is simply can not tell the difference).

2.) I hear GoDaddys servers are pretty full. They may be trying to push people into buying dedicated servers through shady means. It wouldn't be the first time they did something like that.

3.) Tiffany is mentally challenged.

Edit:

Signature related.

I don't think I every read that before, but it is great. I believe I may steal that for a facebook status in the near future.