Delicious      Technorati      Blinklist      Furl      Reddit

TimeTrex Time and Attendance

TimeTrex Time and Attendance


* FAQ    * Search
* Login   * Register

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 7 posts ] 
Author Message
PostPosted: Mon Aug 10, 2015 7:45 am 
Offline

Joined: Wed Jan 16, 2013 2:22 pm
Posts: 29
It appears that the CronJobFactory system built into Timetrex doesn't wait for backup_database to complete before starting new jobs. If backup_database takes longer than a minute, this can cause it to overlap into other jobs.

If backup_database overlaps into the running of PurgeDatabase (in MiscWeekly.php), the spike in load will almost always cause the virtual machine that we are running Timetrex on to lock up.

My solution was to explicitly stop Timetrex cron jobs during a backup window and then run backup_database via my own cron job. This has been working reliably for several weeks.


Top
 Profile  
 
PostPosted: Mon Aug 10, 2015 8:04 am 
Offline

Joined: Tue Sep 19, 2006 2:22 pm
Posts: 7280
When you say "lock-up" what do you mean exactly?

Also what database are you using?


Top
 Profile  
 
PostPosted: Mon Aug 10, 2015 8:23 am 
Offline

Joined: Wed Jan 16, 2013 2:22 pm
Posts: 29
PostgreSQL 8.3.3 on Ubuntu, it's what the Timetrex installer set up. It's on a Dell T110 w/ X3440 CPU, the Timetrex VM is running in Proxmox and is given 4GB ram and a 2 core CPU via KVM32.

When I say lock up, I mean Timerex & Postgre will appear to work normally for a short period of time after the overlapping of jobs happens, but it seems like Postgre can't keep up. Disk access skyrockets and then it eventually locks up itself and it's entire virtual machine. I then need to forcibly kill the Virtual Machine, start it back up, give Postgre a little time to catch up (seems like it's finishing up whatever it was trying to do before, but I'm not sure how to verify that yet), and then start Timetrex like normal.

I do have a capture of timetrex.log, but it doesn't seem to show anything interesting other than the overlapping of jobs.

The RRDtool graphs just show a massive spike around the time it happens (see attachment, lock up happened Sunday).


Attachments:
Untitled.png
Untitled.png [ 51.15 KiB | Viewed 4457 times ]
Top
 Profile  
 
PostPosted: Mon Aug 10, 2015 8:49 am 
Offline

Joined: Tue Sep 19, 2006 2:22 pm
Posts: 7280
The Linux automated installer is primarily designed for extremely old Linux distributions that didn't offer the minimum requirements for TimeTrex, since that hasn't been the case for about 5 years now, we recommend using the manual .ZIP installer and using the bundled versions of Apache/PHP/PostgreSQL that come with your Linux distribution instead. Doing so may actually help alleviate the problem.

The Linux automated installer will be discontinued over the next year or so.


Top
 Profile  
 
PostPosted: Tue Aug 11, 2015 5:53 am 
Offline

Joined: Wed Jan 16, 2013 2:22 pm
Posts: 29
Thanks for the quick reply! I didn't realize the Linux installer had been depreciated.

I set up a test VM running 64 bit Ubuntu with PostgreSQL 9.3.9 & PHP 5.5.9 and am currently restoring a copy of our database.

Assuming everything works well in testing I'll move it into production next week. Then I'll re-enable the backup_database job and report back with what I find out.


Top
 Profile  
 
PostPosted: Thu Sep 03, 2015 11:21 am 
Offline

Joined: Wed Jan 16, 2013 2:22 pm
Posts: 29
Upgrading Postgre to 9.3 and using the tweaks I posted in the installation forums seems to have fixed my problem and made Timetrex run smoother in general.

http://forums.timetrex.com/viewtopic.php?f=1&t=6946

Thanks for the suggestion.


Attachments:
stat.png
stat.png [ 52.02 KiB | Viewed 4349 times ]
Top
 Profile  
 
PostPosted: Thu Sep 10, 2015 4:46 am 
Offline

Joined: Wed Jan 16, 2013 2:22 pm
Posts: 29
Update: It looks like everything has settled down into a nice regular load cycle. I'm not sure why I was getting high loads towards the end of the week initially, but it's gone now.


Attachments:
usage.png
usage.png [ 68.87 KiB | Viewed 4336 times ]
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  


Time and Attendance




Powered by phpBB® Forum Software © phpBB Group