Down for maintence suddently after upgrade.

[JimmyK]

I noticed that the community edition was discontinued.

We've had a system that has worked flawless for some time. A day or so ago, it tried to update and now shows a down for maintenance problem, but I cannot identify the error in the log file.

While I am willing to entertain a paid version, I cannot have a complete disruption with immediately not available. How can I get this to work again?

[P0389604] [00883ms] [L0877]: TimeTrexSoapClient::getPreUpgradeStage(): Writing pre-upgrade stage to disk...
[P0389604] [01227ms] [L0550]: GLOBAL: Running pre-Upgrade Stage... Command: "/usr/bin/php" -d opcache.enable_cli=0 "/var/cache/timetrex/PRE_UPGRADE.PHP" --config "/usr/share/timetrex/html/includes/../timetrex.ini.php" --base_path "/usr/sh
[P0389604] [01227ms] [L0550]: GLOBAL: are/timetrex/html/" Exit Code: 0
[P0389604] [01227ms] [L2117]: Misc::cleanDir(): Cleaning: /var/cache/timetrex Exclude Regex: \.ZIP|\.lock|.state|upgrade_staging
[P0389604] [01227ms] [L1023]: Install::checkPHPVersion(): Comparing with Version: 8.1.26
[P0389604] [01228ms] [L1677]: Install::checkDiskSpace(): Free Space: 19744026624 Free Percent: 58.704625667082
[P0389604] [01228ms] [L1612]: Install::checkPHPOpenBaseDir(): Open BaseDir:
[P0389604] [01228ms] [L2720]: Misc::getCurrentOSUser(): Running as OS User: www-data
[P0389604] [01228ms] [L1244]: Install::checkFilePermissions(): Checking directory readable/writable: /usr/share/timetrex/html Random Num: 119
[P0389604] [01294ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 1000 Current File: /usr/share/timetrex/html/interface/html5/framework/apollo-vue/src/service/NodeService.js
[P0389604] [01362ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 2000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.1/RecurringScheduleControlViewController-CRdMLuCo.bundle.js.map
[P0389604] [01406ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 3000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.7/NoResultBox-S3YfGvw9.bundle.js.map
[P0389604] [01449ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 4000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.3/Form1099NecReportViewController-BKGVhdbq.bundle.js
[P0389604] [01497ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 5000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.0/UserGenericStatusWindowController-B4poA0YR.bundle.js
[P0389604] [01541ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 6000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.8/PayStubSummaryReportViewController-DB1J-Hfd.bundle.js
[P0389604] [01584ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 7000 Current File: /usr/share/timetrex/html/interface/html5/dist/v16.12.2/MyJobApplicationViewController-BU9FIs_f.bundle.js
[P0389604] [01631ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 8000 Current File: /usr/share/timetrex/html/interface/html5/global/widgets/live-chat/
[P0389604] [01709ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 9000 Current File: /usr/share/timetrex/html/vendor/ezyang/htmlpurifier/library/HTMLPurifier/AttrDef/HTML/Nmtokens.php
[P0389604] [01767ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 10000 Current File: /usr/share/timetrex/html/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Calculation/Logical/Conditional.php
[P0389604] [01833ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 11000 Current File: /usr/share/timetrex/html/vendor/pear/pear/tests/PEAR_Installer/test_bug18362.phpt
[P0389604] [01895ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 12000 Current File: /usr/share/timetrex/html/vendor/pear/pear/tests/PEAR_Command_Remote/list-all/test_rest.phpt
[P0389604] [01957ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 13000 Current File: /usr/share/timetrex/html/vendor/kigkonsult/icalcreator/src/Participant.php
[P0389604] [02019ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 14000 Current File: /usr/share/timetrex/html/vendor/phpspec/prophecy/src/Prophecy/Argument/Token/StringContainsToken.php
[P0389604] [02084ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 15000 Current File: /usr/share/timetrex/html/vendor/web-auth/webauthn-lib/src/AuthenticationExtensions/AuthenticationExtension.php
[P0389604] [02148ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 16000 Current File: /usr/share/timetrex/html/vendor/nikic/php-parser/lib/PhpParser/Node/Scalar/MagicConst/File.php
[P0389604] [02201ms] [L1305]: Install::checkFilePermissions(): Batch Completed: 17000 Current File: /usr/share/timetrex/html/classes/modules/accrual/
[P0389604] [02239ms] [L1328]: Install::checkFilePermissions(): Done Checking directory readable/writable: /usr/share/timetrex/html
[P0389604] [02239ms] [L1244]: Install::checkFilePermissions(): Checking directory readable/writable: /var/cache/timetrex Random Num: 100
[P0389604] [02240ms] [L1328]: Install::checkFilePermissions(): Done Checking directory readable/writable: /var/cache/timetrex
[P0389604] [02240ms] [L1244]: Install::checkFilePermissions(): Checking directory readable/writable: /var/log/timetrex Random Num: 119
[P0389604] [02240ms] [L1328]: Install::checkFilePermissions(): Done Checking directory readable/writable: /var/log/timetrex
[P0389604] [02240ms] [L1244]: Install::checkFilePermissions(): Checking directory readable/writable: /var/lib/timetrex/storage Random Num: 141
[P0389604] [02241ms] [L1328]: Install::checkFilePermissions(): Done Checking directory readable/writable: /var/lib/timetrex/storage
[P0389604] [02241ms] [L1338]: Install::checkFilePermissions(): All Files/Directories (17739) are readable/writable! Files Checked: 17739/17739(100%) in: 1s
[P0389604] [02241ms] [L0559]: GLOBAL: New version available, collecting data to download...
[P0389604] [02384ms] [L0603]: TimeTrexSoapClient::sendCompanyData(): Sending Company Data... ID: 11e965be-6679-3580-9b71-0ac3e18dfe8c
[P0389604] [02953ms] [L0731]: TimeTrexSoapClient::sendCompanyData(): Sent Company Data...
[P0389604] [03529ms] [L0459]: TimeTrexSoapClient::sendCompanyVersionData(): Sending Company Version Data...
[P0389604] [03767ms] [L0504]: TimeTrexSoapClient::sendCompanyVersionData(): Sent Company Version Data!
[P0389604] [03767ms] [L0553]: Install::checkTableExists(): Table Name: company
[P0389604] [03770ms] [L0564]: Install::checkTableExists(): Exists - Table Name: company
[P0389604] [04000ms] [L0572] Array: [Function](): File Upgrade URL:
bool(false)


[P0389604] [04000ms] [L0575]: GLOBAL: Upgrade URL not available from server, either already running latest version or not ready to upgrade yet, skip retries: 0
[P0389604] [04000ms] [L0781]: GLOBAL: Upgrade File URL not available...
[P0389604] [04000ms] [L3813]: Factory::StartTransaction(): StartTransaction(): Transaction: Count: 0 Off: 0 OK: 0
[P0389604] [04001ms] [L4214]: Factory::isValid(): Calling Validate() of: SystemSettingListFactory
[P0389604] [04001ms] [L0090] Array: SystemSettingFactory::isUniqueName(): Unique Name: auto_upgrade_failed
string(36) "11e965be-df13-dd30-a256-0ac3e18dfe8c"


[P0389604] [04001ms] [L0306]: SystemSettingFactory::setSystemSetting(): Key: auto_upgrade_failed Value: 1 isNew: 0
[P0389604] [04001ms] [L3813]: Factory::StartTransaction(): StartTransaction(): Transaction: Count: 1 Off: 1 OK: 1
[P0389604] [04001ms] [L4220]: Factory::isValid(): Data has already been validated...
[P0389604] [04001ms] [L4711]: Factory::Save(): Updating ID: 11e965be-df13-dd30-a256-0ac3e18dfe8c
[P0389604] [04005ms] [L4214]: Factory::isValid(): Calling Validate() of: LogFactory
[P0389604] [04005ms] [L3813]: Factory::StartTransaction(): StartTransaction(): Transaction: Count: 1 Off: 2 OK: 1
[P0389604] [04005ms] [L4650]: Factory::Save(): Calling preSave() of: LogFactory
[P0389604] [04005ms] [L4220]: Factory::isValid(): Data has already been validated...
[P0389604] [04005ms] [L4699]: Factory::Save(): Insert ID: 0623f25c-2547-a36e-d474-0ac3e18dfe8c Table: system_log
[P0389604] [04006ms] [L3920]: Factory::CommitTransaction(): CommitTransaction(): Transaction: Count: 1 Off: 3
[P0389604] [04006ms] [L0108]: TTLog::addEntry(): LogDetail Disabled... Object ID: 11e965be-df13-dd30-a256-0ac3e18dfe8c Action ID: 20 Table: system_setting Description: System Setting - Name: auto_upgrade_failed Value: 1 User ID: 00000000-0000-0000-0
[P0389604] [04006ms] [L0108]: TTLog::addEntry(): 000-000000000000
[P0389604] [04006ms] [L4754]: Factory::Save(): Calling postSave() of: SystemSettingListFactory
[P0389604] [04006ms] [L1076]: Factory::removeCache(): Queuing removal of cache: all Group Id: system_setting TrnsCnt: 1
[P0389604] [04006ms] [L1076]: Factory::removeCache(): Queuing removal of cache: auto_upgrade_failed Group Id: system_setting TrnsCnt: 1
[P0389604] [04006ms] [L3920]: Factory::CommitTransaction(): CommitTransaction(): Transaction: Count: 1 Off: 2
[P0389604] [04007ms] [L3915]: Factory::CommitTransaction(): CommitTransaction(): Final Commit... Transaction: Count: 1 Off: 1 OK: 1
[P0389604] [04008ms] [L3858]: Factory::clearCacheRemovedInTransaction(): Removing cached records pending removal during successful transaction: Count: 2
[P0389604] [04008ms] [L1084]: Factory::removeCache(): Removing cache: all Group Id: system_setting
[P0389604] [04008ms] [L1084]: Factory::removeCache(): Removing cache: auto_upgrade_failed Group Id: system_setting
[P0389604] [04008ms] [L0553]: Install::checkTableExists(): Table Name: notification
[P0389604] [04011ms] [L0564]: Install::checkTableExists(): Exists - Table Name: notification
[P0389604] [04011ms] [L0141]: setAutoUpgradeFailed(): Notification table exists, sending notifications about the upgrade failure...
[P0389604] [04013ms] [L0283]: Notification::sendNotificationToAllUsers(): Attempting to send notification to total companies: 1 Type: system
[P0389604] [04025ms] [L0232]: LockFile::readPIDFile(): Lock file exists with PID: 389604 Lock File: /var/cache/timetrex/UnAttended_Upgrade_3250337831.lock
[P0389604] [04025ms] [L0211]: LockFile::delete(): Deleting Lock File: /var/cache/timetrex/UnAttended_Upgrade_3250337831.lock PID: 389604
[P0389604] [04025ms] [L0451]: TTShutdown(): Server Response Time: 4.0248129367828
[P0389604] [04025ms] [L0384]: Debug::showCacheProfile(): Cache Profile: Reads: 25 Hits: 12 (48%) Writes: 13 Deletes: 2
[P0389604] [04025ms] [L0461]: TTShutdown(): Shutting down completely...
---------------[ 1728373695.2892 {08-Oct-2024 3:48:15 -0400} (PID: 389604) ]---------------

[mikeb]

When TimeTrex can't perform critical functions or maintenance jobs and enters a failed state, it will put itself into "Down for Maintenance" mode to prevent further problems from occurring. So this is often a consequence of a larger problem.

I would recommend upgrading to the latest version of TimeTrex (v16.12.9) first, then edit your timetrex.ini.php and change:

Code: Select all

down_for_maintenance = TRUE

to

Code: Select all

down_for_maintenance = FALSE

If your system goes into maintenance mode again after that, attach a full copy of your log file here so we can investigate further.

[JimmyK]

Thank you,

I've never performed a manual upgrade, we've only allowed the automatic upgrades to take place. Is there a document somewhere that outlines the upgrade procedure for the ubunti/debian version?

It does look like the upgrade started failing toward the end of September.

For now I changed down_for_maintenance = FALSE and it immediately changed back to down_for_maintenance = TRUE but the log shows another upgrade attempt. if I change down_for_maintenance = TRUE and add in the command to disable automatic upgrades it will stay running and not go into maintenance mode.

It seems like the upgrade fails, but normally there is some clear reason why, for example, when our PHP version was too low.

[mikeb]

Its likely that your system does not meet the minimum requirements of both PHP or PostgreSQL as listed in the below link, therefore the upgrade is failing, or even your current installation is failing, which is putting the system into maintenance mode.
viewtopic.php?t=6

You could try forcing the automatic upgrader to run, but if the minimum requirements are not met, it will just fail again, the command to do that is this:

Code: Select all

php tools/unattended_upgrade.php -f

A full manual upgrade is simple, just download the TimeTrex .ZIP file, and unzip it over top of your existing TimeTrex directory, then open the web installer (/interface/install/install.php) to complete the upgrade process.

[JimmyK]

Thank you for the tips.

I’ll check and see if there are any upgrades needed to either of those and retry the upgrade. If that fails I’ll try to do the instal. My concern was the new install just wiping the database but it sounds like that’s not the case.

I know we upgraded to PHP 8 in the past but I don’t recall upgrading PostgreSQL ever.

The OS itself will get updates but I don’t know that it updates SQL on its own.

[JimmyK]

Forcing the automatic upgrade to run allowed the upgrade to proceed.

It did set it back to maintenance mode afterwards, but I changed it back to false. I left the automatic upgrade off, since presumably there won't be anymore.

PostgreSQL is at version 12 and up to date, PHP is up to date. I did run some OS updates that had been behind by about 6 months.

If it goes back to maintenance mode, I will post log files.

Thanks for all the years of supporting a great product in the community edition setting. It's sad to see product after product go this direction, but it's understandable.