Recently we had some cases where AppFabric caching service was crashing with error "An entry with the same key already exists."

Issue: In the AppFabric event logs (Applications and Services Logs > Microsoft > Windows > Application Server-System Services), we'll see below error logged.

AppFabric Caching service crashed with exception {System.ArgumentException: An entry with the same key already exists.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
at Microsoft.Fabric.Data.PartitionTable.UpdateEntry(LookupTableEntry newEntry)
at Microsoft.Fabric.Data.PM.PMPartitionTable..ctor(PartitionManager pm, IList`1 partitions, Int64 savedVersion, Object lockObject)
at Microsoft.Fabric.Data.PM.PartitionCache..ctor(PartitionManager pm, IPartitionManagerStore pmStore, LoadTable loadTable, IList`1 partitions, Int64 savedLookupVersion)
at Microsoft.Fabric.Data.PM.PartitionManager.ProcessLoadPM(Object state)
at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.Invoke(WaitCallback callback, Object state)
at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.WorkerThreadStart()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()}. Check debug log for more information

Reason: This is happening at PartitionTable.UpdateEntry, which indicates a problem with the cache partitions.

Solution:

We would need to wipe the partitions out and rebuild them.

-- Exported and Imported cache cluster config…
1. Open PowerShell as Administrator.
2. Use-CacheCluster.
3. Export-CacheClusterConfig c:\output.txt
4. Open output.txt with Notepad.
5. Find and replace <dataCache size=”Medium”> to <dataCache size=”Small”>.
6. Find and replace <caches partitionCount=”256″> to <caches partitionCount=”32″>.
7. Save and close output.txt.
8. Go back to PowerShell as Administrator.
9. Run: Stop-CacheCluster to ensure that AppFabric Distributed Cache Service is not running.
10. Run: Import-CacheClusterConfig c:\output.txt.
11. Run: Start-CacheCluster and ensure that all cache hosts are started.
12. Wait a few minutes to allow the cache hosts to start completely.

Select one of the following three options to optimize performance based on cluster size. This setting is available only if the New Cluster control is selected.

• Small [1-5 computers]
• Medium [6-15 computers]
• Large [> 15 computers]

Reference link: https://msdn.microsoft.com/en-us/library/ff637694%28v=azure.10%29.aspx?f=255&MSPPError=-2147217396

Hope this information helps!

Rohit Soni (MSFT)

Issue: We would not be able to deserialize powershell object (PSObject) using WCF Message class directly.

Error:

"The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://schemas.xyz.com/Services/MyService:DoServiceResult. The InnerException message was 'Member 'CliXml' was not found.'.".

• Cause –
Failing Message:
<CliXml i:type="d3p1:string" xmlns:d3p1="http://www.w3.org/2001/XMLSchema">&lt;Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"&gt;

Working Message:
<CliXml xmlns:d7p1="http://www.w3.org/2001/XMLSchema" i:type="d7p1:string" xmlns="">&lt;Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"&gt;

Working message defines the namespace for the CliXml element where the failing version does not.

• Resolution –

We added a message inspector on the WCF services (not the client) and inspected the message in BeforeSendReply. It can also be compare by taking a service side WCF traces.

A simple workaround for this is to make a change similar to this:

MessageConverter.cs:

public Message ResponseToMessage(ResponseBase response)
{
try
{
var sb = new StringBuilder();

using (var sw = new StringWriter(sb))
using (var tw = new XmlTextWriter(sw))
{
var serializer = new DataContractSerializer(response.GetType(), BodyInfo.Operation + "Result", String.Empty);
serializer.WriteStartObject(tw, response);

tw.WriteAttributeString("xmlns", "b", null, OperationInfo.Service.Namespaces.Base); // "base" namespace
tw.WriteAttributeString("xmlns", "e", null, OperationInfo.Namespaces.Response ); // "entity" namespace

serializer.WriteObjectContent(tw, response);
serializer.WriteEndObject(tw);
}

var responseBody = String.Format("<{0} xmlns=\"{1}\">{2}</{0}>", BodyInfo.Operation + "Response", OperationInfo.Service.Namespaces.Service, sb.ToString());
responseBody = responseBody.Replace("<CliXml ", "<CliXml xmlns=\"\" ");
XmlDocument doc = new XmlDocument();
doc.Load(new StringReader(responseBody));

var responseAction = OperationInfo.Service.Namespaces.ReplyAction + BodyInfo.Operation + "Response";
return Message.CreateMessage(SoapMessage.Version, responseAction, new XmlNodeReader(doc));
}
catch (Exception ex)
{
throw new ApplicationException(String.Format("ResponseToMessage exception: '{0}'!", ex.Message), ex);
}
}

With this only change on the server side “ServiceHosting”, the “client” is able to read the “PSObject” information de-serialized perfectly fine.

Rohit Soni (MSFT)

WCF PII Verbose tracing
====

Depending on app built, update the machine.config file

Add line:
<machineSettings enableLoggingKnownPii="true"/>

<!-- Sample -->
<system.serviceModel>
<machineSettings enableLoggingKnownPii="true"/>

OUTSIDE <System.ServiceModel> tag add below lines:

<system.diagnostics>
<sources>
<source logKnownPii="true" name="System.ServiceModel.MessageLogging"
switchValue="Verbose,ActivityTracing">
<listeners>
<add type="System.Diagnostics.DefaultTraceListener" name="Default">
<filter type="" />
</add>
<add name="ServiceModelMessageLoggingListener">
<filter type="" />
</add>
</listeners>
</source>
<source propagateActivity="true" name="System.ServiceModel" switchValue="Verbose,ActivityTracing">
<listeners>
<add type="System.Diagnostics.DefaultTraceListener" name="Default">
<filter type="" />
</add>
<add name="ServiceModelTraceListener">
<filter type="" />
</add>
</listeners>
</source>
</sources>
<sharedListeners>
<add initializeData="c:\temp\app_messages.svclog"
type="System.Diagnostics.XmlWriterTraceListener, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
name="ServiceModelMessageLoggingListener" traceOutputOptions="LogicalOperationStack, DateTime, Timestamp, ProcessId, ThreadId, Callstack">
<filter type="" />
</add>
<add initializeData="c:\temp\app_tracelog.svclog"
type="System.Diagnostics.XmlWriterTraceListener, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
name="ServiceModelTraceListener" traceOutputOptions="LogicalOperationStack, DateTime, Timestamp, ProcessId, ThreadId, Callstack">
<filter type="" />
</add>
</sharedListeners>
</system.diagnostics>

INSIDE <System.ServiceModel> tag add below lines:

<system.serviceModel>
<diagnostics>
<messageLogging logEntireMessage="true" logKnownPii="true" logMalformedMessages="true"
logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" />
<endToEndTracing propagateActivity="true" activityTracing="true"
messageFlowTracing="true" />
</diagnostics>

ONCE DONE, we should see following entry in Event logs
============
PII logging has been turned on. Sensitive information will be logged in the clear.
Process Name: ConsoleApplication1.vshost
Process ID: 5996

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="System.ServiceModel 4.0.0.0" />
<EventID Qualifiers="49153">107</EventID>
<Level>4</Level>
<Task>7</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2018-02-06T00:56:31.835601400Z" />
<EventRecordID>133301</EventRecordID>
<Channel>Application</Channel>
<Computer>CC.XX.XXX.XXX</Computer>
<Security UserID="XXX.XXX" />
</System>
- <EventData>
<Data>ConsoleApplication1.vshost</Data>
<Data>5996</Data>
</EventData>
</Event>

I hope this helps

Thanks

SS

I had a customer who has a claim aware ASP.NET web application hosted on two IIS servers behind a load balancer.

Scenario:

They had a requirement where they would want to stop (bring down) server1 to perform some patch or maintenance activity. However, they did not want the users are being served by the affected server (server1) and use the same cookies that are set by server1 so that the load balancer would redirect the request to server2 and work seamlessly.

For testing this, we made a request to the web app from a client machine. We know whether the requested reached either Server1 or Server2 from IIS logs. Let say the request was served by Server1. We manually went to the IIS sever1 and stopped the application corresponding to the application.

On the client, when navigated to a different page with logged in session, LB identified the Server1 is down and redirected the request to server2.

Then, we ran into the following exception:

General Approach:

We have a couple of blogs with detailed explanation on this issue and how to resolve it

• https://blogs.msdn.microsoft.com/distributedservices/2012/10/29/wif-1-0-id1073-a-cryptographicexception-occurred-when-attempting-to-decrypt-the-cookie-using-the-protecteddata-api/
• https://blogs.msdn.microsoft.com/napegadie_kones_msft_blog/2016/01/19/cryptographicexception-key-not-valid-for-use-in-specified-state/

This exception usually occurs, when there is a load balancer that routes incoming user requests to one of the servers in a cluster environment.  In other words, the exception will occur if the request gets routed to a different server than the one that served the first request for that session.

There are three ways to resolve this issue as mentioned in the above blogs:

1- Use sticky sessions at the Load Balancer

2- Use X509 Certificate to secure sessions cookies

3- Use DPAPI with matching machine keys

Challenges:

In our scenario, enabling sticky sessions will not help. And we don’t want to use certificate since it needs some code changes. We are only left with option 3 where use matching machine keys across the machines in the cluster for encrypting and decrypting session cookies.

Though we generate machine keys from one IIS machine and copied to all the servers in the load balancer, we might still see it is failing with the same error.

By adding matching Machines keys to the web.cofig or machine.config will not suffice.

WIF provides a specialized session token handler, the MachineKeySessionSecurityTokenHandler, that protects tokens by using the keys specified in the <machineKey> element.

Configure your application to use MachineKeySessionSecurityTokenHandler by adding it to the token handler collection. Remove  SessionSecurityTokenHandler, and add MachineKeySessionSecurityTokenHandler.

I added the following in config:

<securityTokenHandlers>

    <remove type="SessionSecurityTokenHandler"/> 

    <add type="MachineKeySessionSecurityTokenHandler" /> 

</securityTokenHandlers>

With the above config, ran into configuration errors that it could not find the specified handler, and the site itself doesn’t load.

Then, I modified the config the following way:

<securityTokenHandlers>

 <clear/>

    <remove type="System.IdentityModel.Tokens.SessionSecurityTokenHandler, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

    <add type="System.IdentityModel.Services.Tokens.MachineKeySessionSecurityTokenHandler, System.IdentityModel.Services, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

</securityTokenHandlers>

Then, I ran into the following error:

Solution:

While I edited the config using visual studio editor, <clear/> tag got added within section. And encountered the above error. Make sure tag is removed.

The right config element that worked is the following:

<System.IdentityModel>
     <identityConfiguration>
          <securityTokenHandlers>
               <remove type="System.IdentityModel.Tokens.SessionSecurityTokenHandler, System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
               <add type="System.IdentityModel.Services.Tokens.MachineKeySessionSecurityTokenHandler, System.IdentityModel.Services, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
          </securityTokenHandlers>
     </identityConfiguration>
</System.IdentityModel>

Hope this helps!