Cyclos logs eating up all disk space

[atitokor]

Hello Admins

My logs are filling up so fast consuming the entire disk space that I have to be clearing them every few hours to keep the site running. Already I have doubled my disk space in 24 hours but still that doesn't help. I'm using Cyclos 4.12 on Docker and the issue is related to the Cyclos container.

Below is a most recent segment of the log.

Thanks in advance for your attention.

Art

Code: Select all

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-06-24T20:57:59.209Z WARN PurgeUnconfirmedUsersPollingTask - Error when Expired pending users purging would mark the given entity as processed: User#10: Mubasher M
org.cyclos.model.IllegalActionException: null
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1507) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.CRUDServiceImpl.remove(CRUDServiceImpl.java:908) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.CRUDServiceImpl.remove(CRUDServiceImpl.java:264) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.PurgeUnconfirmedUsersPollingTask.process(PurgeUnconfirmedUsersPollingTask.java:90) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.PurgeUnconfirmedUsersPollingTask.process(PurgeUnconfirmedUsersPollingTask.java:1) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.lambda$3(EntityBatchPollingTask.java:237) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.runEnsuringInvocationContext(TransactionHandlerImpl.java:183) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.doRun(TransactionHandlerImpl.java:109) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.run(TransactionHandlerImpl.java:159) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.run(TransactionHandlerImpl.java:80) ~[cyclos-impl.jar:4.12]
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at com.sun.proxy.$Proxy34.run(Unknown Source) ~[?:?]
        at org.cyclos.impl.InvokerHandlerImpl.performInTransaction(InvokerHandlerImpl.java:238) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.InvokerHandlerImpl.doRunAsInTransaction(InvokerHandlerImpl.java:187) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.InvokerHandlerImpl.runAsInTransaction(InvokerHandlerImpl.java:168) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.doProcessAndFinish(EntityBatchPollingTask.java:214) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.doRun(EntityBatchPollingTask.java:81) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.AbstractPollingTask.run(AbstractPollingTask.java:77) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.cluster.AbstractClusterHandlerImpl.runPollingTask(AbstractClusterHandlerImpl.java:153) ~[cyclos-impl.jar:4.12]
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.cyclos.bootstrap.BeanHandlerImpl$CustomBeanInvocationHandler.invoke(BeanHandlerImpl.java:84) ~[cyclos-impl.jar:4.12]
        at com.sun.proxy.$Proxy32.runPollingTask(Unknown Source) ~[?:?]
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at com.sun.proxy.$Proxy42.runPollingTask(Unknown Source) ~[?:?]
        at org.cyclos.impl.utils.tasks.PollingTaskHandlerImpl.lambda$1(PollingTaskHandlerImpl.java:184) ~[cyclos-impl.jar:4.12]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-06-24T20:57:59.213Z ERROR PurgeUnconfirmedUsersPollingTask - Error on task Expired pending users purging
org.cyclos.model.IllegalActionException: null
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1507) ~[cyclos-impl.jar:4.12]

[luis]

For some reason, it seems like you have inconsistent data in the users table.
There is a task that keeps reading users which haven't yet confirmed the registration (via e-mail validation) after some time.
As a quick workaround, disable that by setting the following in cyclos.properties:

Code: Select all

cyclos.purgeUnconfirmedUsers.days = 0

To help us understand the issue, please send here the result of the following SQL query:

Code: Select all

select name, registration_confirmation_date, validation_key_type, status from users where id = 10;

We've made sure to fix this for the next version, 4.12.1.

[atitokor]

Thanks! I updated cyclos.properties and below are the results of the sql query. I do confirm that I removed this user on 6/23 but the issue has been there before then.

Code: Select all

name	Mubasher M
registration_confirmation_date	NULL
validation_key_type	REGISTRATION
status   REMOVED

Sure! Will update to 4.12.1 as soon as it's out

Again, thanks so much for your time and attention.

[luis]

Ok, thanks for the info.
If the issue was happening before, it might have other users that were removed before being confirmed, and that is what caused the bug.
After 4.12.1 you can revert the change in cyclos.properties if you wish.