Help needed for exception

Posts   
 
    
AlexanderM
User
Posts: 41
Joined: 18-May-2012
# Posted on: 03-Mar-2015 13:58:47   

Hello,

What might cause the following exception with stacktrace in LlblGen? I think this is not a problem in LlblGen but I'm looking for possible causes so you're help is very much appreciated. My line of code is executed thousands of times with no problems but sometimes crashes with this error on different environments (Test, Acceptation, Production). I guess I'm interested in what happens in D.LLBLGen.Pro.LinqSupportClasses.LinqUtils.ConvertLocalExpressionToRealObject.

Btw, We're using version 4.2.14.811.

Regards Alexander.


System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at SD.LLBLGen.Pro.LinqSupportClasses.LinqUtils.ConvertLocalExpressionToRealObject(Expression toConvert)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleInMemoryEvalCandidateExpression(InMemoryEvalCandidateExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleUnaryExpression(UnaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleUnaryExpression(UnaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleUnaryConvertExpression(UnaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpressionList(ReadOnlyCollection`1 listToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleDefaultOperatorOverloadMethod(MethodCallExpression expressionToHandle, StandardOperatorType operatorType)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleMethodCallExpression(MethodCallExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleBinaryExpression(BinaryExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleLambdaExpression(LambdaExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleMethodCallWhere(MethodCallExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleQueryableExtensionMethod(MethodCallExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleMethodCallPerType(MethodCallExpression expressionToHandle, Type declaringType)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.PreProcessor.HandleMethodCallExpression(MethodCallExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.SD.LLBLGen.Pro.LinqSupportClasses.ILLBLGenProQuery.Execute[TResult]()

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39590
Joined: 17-Aug-2003
# Posted on: 03-Mar-2015 17:35:31   

It compiles the lambda (likely one in the projection) into a delegate and then invokes that delegate, which then will result in a value which is used in the query. this is done with method calls to local methods or access to objects local to the query. the invoke here causes an exception in the object invoked. What that is, is unclear though. Could you post the query where this happens?

Frans Bouma | Lead developer LLBLGen Pro
AlexanderM
User
Posts: 41
Joined: 18-May-2012
# Posted on: 04-Mar-2015 10:27:09   

One example is this. I have two other places where the same poblem occurs but they are alike. All three places are about reading a queue, with a retry mechanism for a certain amount of days. If sentcounter < 5 we do it every time (minute) the method is called otherwise eevery 30 minutes.


IQueryable<SmsQueueEntity> query = _metaData.SmsQueue;
int daysInPast = Config.QueueSMSRetryDurationDays;
query = query.Where(x => x.CompletedTimestamp == null && x.CreationTimestamp >= DateTime.Now.AddDays(-daysInPast) && ((x.SentCounter == null || x.SentCounter < 5) || x.LastChangeDate < DateTime.Now.AddMinutes(-30)));
resultSet = ((ILLBLGenProQuery)query).Execute<IEnumerable<SmsQueueEntity>>();

Some extra info: The last three months (first time on 4 dec) I have seen this error 19 times on 4 different environments on all 5 servers in these environments. It has not happened in PROD but because it does not happen very often, and PROD got the code version on 3 feb, it's too soon to know if it won't happen on PROD.

Furthermore this code runs in a scheduler named Quartz.NET. The big problem is that I catch the error, throw a new Exception for Quartz.NET (as documented), Quartz.NET logs the exception and my application stops with an unhandled exception, with a stacktrace with this error in it. I checked the code of Quartz.NET (open source) and it catches the error as documented, so I don't understand why my application crashes. Other errors that are handled this way (e.g. when they shutdown our database) behave as predicted, indicating the error mechanism is good in "normal" conditions.

So I'm investigating two questions: 1) Why does LlblGen in very rare cases run into this error? 2) Why does my application crash, so what is special about this error?

One extra question for you: Can you give a scenario how I can create this Exception on purpose, so I can reproduce it in my DEV environment to investigate the way the exception is handled?

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39590
Joined: 17-Aug-2003
# Posted on: 04-Mar-2015 17:14:54   

AlexanderM wrote:

One example is this. I have two other places where the same poblem occurs but they are alike. All three places are about reading a queue, with a retry mechanism for a certain amount of days. If sentcounter < 5 we do it every time (minute) the method is called otherwise eevery 30 minutes.


IQueryable<SmsQueueEntity> query = _metaData.SmsQueue;
int daysInPast = Config.QueueSMSRetryDurationDays;
query = query.Where(x => x.CompletedTimestamp == null && x.CreationTimestamp >= DateTime.Now.AddDays(-daysInPast) && ((x.SentCounter == null || x.SentCounter < 5) || x.LastChangeDate < DateTime.Now.AddMinutes(-30)));
resultSet = ((ILLBLGenProQuery)query).Execute<IEnumerable<SmsQueueEntity>>();

Some extra info: The last three months (first time on 4 dec) I have seen this error 19 times on 4 different environments on all 5 servers in these environments. It has not happened in PROD but because it does not happen very often, and PROD got the code version on 3 feb, it's too soon to know if it won't happen on PROD.

Furthermore this code runs in a scheduler named Quartz.NET. The big problem is that I catch the error, throw a new Exception for Quartz.NET (as documented), Quartz.NET logs the exception and my application stops with an unhandled exception, with a stacktrace with this error in it. I checked the code of Quartz.NET (open source) and it catches the error as documented, so I don't understand why my application crashes. Other errors that are handled this way (e.g. when they shutdown our database) behave as predicted, indicating the error mechanism is good in "normal" conditions.

So I'm investigating two questions: 1) Why does LlblGen in very rare cases run into this error?

I have no idea, it's not our error. We do this:


/// <summary>
/// Converts the local expression to a real object. This is done by a trick: we'll compile the whole expression as a lambda and run it. 
/// this will return the object, including initialization results and completely materialized as we want it. The passed in expression is for example a 
/// list or array initialization / ctor call. To utilize the in-memory object, we've to run this ctor to be able to use the values. This routine takes care 
/// of that.
/// </summary>
/// <param name="toConvert">To convert.</param>
/// <returns>The result of the expression after running it wrapped in a ConstantExpression. </returns>
public static ConstantExpression ConvertLocalExpressionToRealObject(LinqExpression toConvert)
{
    return LinqExpression.Constant(LinqExpression.Lambda(toConvert, new ParameterExpression[0]).Compile().DynamicInvoke());
}

Here 'toConvert' is an expression which has no ties to any db element. Looking at the stack trace, the call crashed in the where method call, and I see you refer to a local variable there (daysInPast). This ends up as a ConstantExpression where <>displayClass or something like that is referenced as the object containing daysInPast (enable linq tracing to level 3 to see the expression tree in text, you'll see the reference there).

To obtain the value for daysInPast the linq provider will then compile the local expression to ()=>localexpression, and run it. It will then return the value daysInPast represents.

I have no idea why this fails in some cases, to be honest.

2) Why does my application crash, so what is special about this error?

The error suggests that when calling the compiled lambda (so the DynamicInvoke call in the method above) it ran into an error. What is strange is that it looks like it can't obtain the local value, which is odd considering the query is in the same scope, so the outer object should still exist.

One extra question for you: Can you give a scenario how I can create this Exception on purpose, so I can reproduce it in my DEV environment to investigate the way the exception is handled?

If I do this:


private static string Throw()
{
    throw new NotImplementedException();
}


[Test]
public void ThrowInMethodInWhereTest()
{
    var metaData = new LinqMetaData(new DataAccessAdapter());
    var q = metaData.Customer.Where(c => c.CustomerId == Throw());
    foreach(var v in q)
    {

    }
}

It crashes at the same spot. There's a difference though: this crash has a different stacktrace, as it's all managed code. It seems that the crash you run into isn't due to a .NET exception, but due to unmanaged code that doesn't play nice.

Frans Bouma | Lead developer LLBLGen Pro
AlexanderM
User
Posts: 41
Joined: 18-May-2012
# Posted on: 20-Mar-2015 13:46:24   

Thanks for your information. We suspect it's memory pressure. We have 2GB memory in our NON-prod servers and we see approx. 1,3GB used. But there's one general windows service process which has a peak memory usage of 1,2 GB but almost always uses 100MB. So our theory is that when this process starts to use it's peak memory we run out of memory and we see these errors which behave different as expected under normal circumstances.