|
|
Browse by Tags
All Tags » Networking (RSS)
-
Simplicity is elusive. A few weeks ago I learned that part of transaction flow, propagating information about the source machine between the client and server, is more complicated than I thought. It's not that the details were inherently complicated but rather that they were inconsistent. The information was passed in a certain header of the message, except when using a particular transaction protocol and transport protocol together. Someone noticed that a few bytes could be saved by optimizing this particular combination and in that case put the information in a different header in a different format. Efficiency was achieved at the cost of forcing everyone trying to understand the protocol to think harder about it. Most of the protocols that have stood the test of time on the web have sacrificed efficiency to achieve simplicity. Consider all of the protocols out there that are large, bloated, and redundant but easy to think about and tolerant to misunderstandings. What keeps them alive against their slimmer and more advanced competition other than that they were able to attract many people to speak them? Next time: Naming Contracts for Versioning Read More...
|
-
Earlier this week a new beta release came out for Network Monitor. Although the beta has a variety of new features, the one I'm most interested in is the change to the capture engine to reduce the number of dropped frames. I've had problems in the past with missing frames in Network Monitor captures, especially for fast networks when there is a substantial CPU load. You can get the release from the Network Monitor Connect site . Read More...
|
-
Many common networking problems can be diagnosed by tracing System.Net events. This is often much easier than setting up packet captures or other software, particularly if you're working on a production machine. Since tracing is a part of the framework, it works almost everywhere just by dropping some additional configuration. Here's an example of a complete configuration file for turning on System.Net tracing. < configuration > < system.diagnostics > < trace autoflush ="true" /> < sources > < source name ="System.Net" > < listeners > < add name ="MyTrace" /> </ listeners > </ source > </ sources > < sharedListeners > < add name ="MyTrace" type ="System.Diagnostics.TextWriterTraceListener" initializeData ="trace.txt" /> </ sharedListeners > < switches > < add name ="System.Net" value ="Verbose" /> </ switches > </ system.diagnostics > </ configuration > You can also make the event source more specific by changing System.Net to one of System.Net.Sockets, System.Net.Cache, or System.Net.HttpListener. Next time: You Are Here Read More...
|
-
As I mentioned on the 30th anniversary of IP , an early and fundamental split was made in TCP to distinguish point-to-point messaging from end-to-end messaging. The split is based on a philosophy that the communication endpoints should be in control and that there should be a minimal amount of functionality and responsibility distributed along the communication path in between. The end-to-end philosophy shows up in how TCP systems perform throttling. Duplication of throttling makes the system more complicated. Duplication of throttling also makes the system less efficient because more throttling decisions have to be made by intermediaries that have less information about the system than the endpoints. TCP uses flow control to establish how fast data can be transmitted from the sender to the receiver. Rather than having the communication path adjust the flow rate, TCP uses congestion control algorithms to infer how the sender should change its transmission strategy to best use the network. In the basic implementation of TCP, the data stream is overlaid by a transmission window that defines which packets can be sent. As packets are received and acknowledged, the transmission window slides along the data stream. The endpoints grow and shrink the size of the transmission window as needed to adjust the flow rate. Initially the transmission window is very small because the state of the communication path is unknown. However, the window size adapts itself over time to an optimal rate of transmission. Some of the changes to the TCP/IP stack in Windows Vista were to accelerate the rate of adaptation and reduce the amount of time spent using a sub-optimal transmission rate. The end-to-end philosophy also shows up in how the WCF TCP channel avoids trying to do throttling. The service knows more about how it's going to use the network than the networking stack does. Consequently, if you want to control how quickly things are sent and received, the best place to control that is in the application. The only bottleneck in the TCP channel is how quickly it can pass information back and forth between the application and the wire. The default quota settings for the channel are sufficient for networks up to 100 Mbps. You'll want to increase the ConnectionBufferSize quota from its default of 8 KB to something like 64 KB on faster networks. Note that if the throttling is desired for fairness between applications rather than within an application, then the best place to do the throttling Read More...
|
-
Today's post is more observational than informational. Enjoy. It's sometimes possible to write XML without having an XML library. If your XML documents are sufficiently similar and templated, then you can craft validly formed XML through little more than string manipulation. The trivial case is where the string is a constant expression and the XML document is actually the same every time through. The more interesting case is where the string has many spots where you can fill-in-the-blank with arbitrary content. Once that content becomes constrained instead of arbitrary, you'll almost always want to make use of an XML library assuming that you want your emitted messages to be validated. It's pretty much impossible to read XML without having an XML library. XML simply has too many rules about whitespace handling, processing instructions, character formats, and nested elements to realistically build hand-crafted parsers in an application. A hand-crafted parser likely supports exactly the message that you saw one day looking at the wire but contains numerous bugs that prevent all of the equivalent ways of writing the same message from being accepted. Even seemingly unimportant changes to the network configuration can result in literal changes, such as the text encoding or the division of text into text nodes expected by a DOM. If you want to understand XML, then you'll need to get a real XML parser. Using an XML parser instead of string manipulation is more expensive but if you can't afford this cost, then you probably can't afford the benefits that XML provides either. Next time: Accessing the Query String Read More...
|
-
On January 1st, 1983 the ARPANET officially switched over from using NCP (Network Control Protocol) to TCP/IP (Transmission Control Protocol over Internet Protocol). This followed a year where the ARPANET supported a mix of NCP and TCP/IP machines using relay servers that supplied Telnet, FTP, and mail services across the two protocols. While the Telnet and FTP services were very similar between the two protocols, mail service was much improved thanks to the recently developed SMTP standard. Before that, mail messages were moved around using a protocol on top of FTP. Although NCP only lasted about 12 years, TCP/IP has lasted more than twice as long with no signs of going away any time soon. Read More...
|
-
There needs to be some concept of cleanup that takes place when an asynchronous request can't be completed. For example, when a service is shut down or a socket is closed you know that any asynchronous operation waiting on that resource will never get a result. Something needs to happen to the asynchronous operations to prevent them from lingering forever. The asynchronous callback pattern has no built-in concept for cancelling a request that you made previously. Even if you build in the concept of cancellation, by its fundamental nature an asynchronous call is going to race to completion against any actions you take. It would be possible to examine a request, see that it hasn't been completed, attempt to cancel the request, and then find out that it completed anyway because completion occurred during that same interval of time. Consequently, cleanup has to be initiated by the request itself rather than by the caller. There are two typical patterns that you see asynchronous operations implement to perform cleanup. The fencepost pattern is where the operation cleans up by completing successfully but giving a distinguished value that indicates that the operation had no result, such as by returning a null object. The exception pattern is where the operation cleans up by storing an exception caught on the worker thread, signaling completion, and then throwing the stored exception on the user thread that goes to pick up the result. The fencepost pattern is typically used for expected cases, such as shutting down, whereas the exception pattern is typically used for unexpected cases, such as IO failures. As an example, consider a service waiting for incoming client connections on a socket. The service will keep several asynchronous requests ready at a time waiting to accept clients. When the socket is shut down, those asynchronous requests need to be cleaned up and they'll do that by returning null. On the other hand, if the socket had encountered a read error instead, then cleanup would have been done by throwing an exception for each outstanding request. Next time: Controlling HTTP Connection Limits Read More...
|
-
Being thread-safe is different than being concurrent. The channel interfaces are thread-safe so that multiple callers can use them at the same time without getting garbled messages. However, if multiple callers try to send messages on a single channel at the same time then a few different things might happen. The channel might be fully concurrent and process the sends in an overlapped fashion. The channel might be partially concurrent and process a limited number of the sends at a time. The other sends are blocked until one of the earlier requests completes. The channel might be singly concurrent and process the sends in a sequential fashion. A connection-oriented channel like TCP tends to be singly concurrent. Interleaving messages requires having a sophisticated framing protocol and the performance cost is rarely worth handling this special case. Datagram channels on the other hand tend to be at least partially concurrent. For example, HTTP in theory is fully concurrent to any degree that you'd like but the HTTP specification recommends limiting clients to two connections to any particular server, resulting in partial concurrency. Whenever you have less than full concurrency you run the risk of deadlock with certain application designs. That's because the logical operation order of the application might say that caller 1 must complete a send before caller 2 can complete, but the physical operation order of the channel says that caller 1 cannot start a send before caller 2 completes. I've seen customers encounter this problem when using HTTP and callbacks. The original request is waiting for the callback operation to complete but the callback thread cannot make any progress because all of the connections are tied up by existing callers, such as the original request. This is a case where the performance gain of sharing connections has led to unsafe behavior. The solution is to increase the concurrency of the system so that at least one call from any of the ongoing chains of operations can complete (there can be callbacks on top of callbacks in complicated knots so the problem is not as simple to solve as increasing the concurrency factor to two). Concurrency can be increased directly at the networking level or by isolating networking resources into separate pools at the application level. Next time: Windows and UPN Format Credentials Read More...
|
-
How do I detect when the other side of a TCP connection has gone away? Does TCP keep-alive take care of this for me? Although we take it for granted that change can be quickly detected for closely connected components, it turns out to be surprisingly difficult to detect change when two machines are isolated by more than a simple wire. Even a really big change to the system, like one of the machines disappearing, is hard to spot. Detecting that the other side has disappeared is a common request because, on a server, knowing that the client has dropped the connection allows you to clean up resources much faster. The TCP transport sometimes gives you quick notifications by aborting the session that the connection has been dropped. However, there's no guarantee that the transport will be able to detect that the other side has gone away. That's because notification of a TCP connection reset has to travel just like any other piece of data and can be lost or redirected along the way, if it was sent at all. The only sure thing is that the next time you attempt an IO operation, you'll find out if the channel was still good or not. If you're unhappy waiting for the next IO operation, then you can make IO operations happen faster. The basic concept is to have a cheap IO operation that does nothing but bounce between the two parties. This is sometimes called a heartbeat and is exactly what takes place when you talk about TCP keep-alive. However, the standard keep-alive interval for TCP is 120 minutes, which is probably worse than your current latency for detecting change. By default, a service gets bored waiting after about 10 minutes and gives up. The chance of a keep-alive happening between the time that a client disconnects and the service notices it is pretty small. If you want something faster but don't want to change the timeouts, then you can take the basic concept into your own hands. One approach is to create a keep-alive method on your service contract that does nothing but let you trigger IO operations at a frequency you desire. Another approach is if you control both ends and don't want to change your service contract, then you can do the same thing in a protocol channel and swallow those messages so that the service never has to see them. Next time: Collections without CollectionDataContract Read More...
|
-
Exciting developments in the world of network traffic capture and protocol analysis. Microsoft Network Monitor has a new companion program that lets you take network captures without having to install any software ahead of time. It even runs off of a file share or USB key. I've been using it recently to collect traces when people ask me to help figure out why their web service doesn't work. The program is still experimental but you can get a copy by joining Microsoft Connect. If you go to the available connections page , sign in, and pick the Network Monitor 3 connection, you'll be able to get the One Click Capture beta from the Downloads section. You'll still want a normal installation of Network Monitor to look at the capture output but you can do that on your machine without having to install the software everywhere you want to take a capture. Next time: Controlling for Clock Skew Read More...
|
-
How can I use reliable messaging together with a load balancer? The point of reliable messaging is to help ensure that messages get from one place to another. This means that the protocol notices when messages that were expected to be delivered go missing. On the other hand, the point of a load balancer is to make sure that messages are spread out and that there aren't too many messages going to the same place. You can see how these two goals might come into conflict at times. Many load balancers do offer a compromise to make these two goals simultaneously achievable though. A perfect load balancer would take the total sum of messages delivered and divide those messages evenly up among the available processing nodes. However, assume that the number of messages in any one client session is relatively small compared to the total number of messages processed. A load balancer could instead apportion groups of messages among the available processing nodes where the total sum of messages is still split roughly equally. This division according to groups would allow a feature like reliable messaging to work because the same server would be used to process all of the messages in the reliable session. The feature that this division method represents is typically called "sticky sessions" or some other phrase for affinitization in the load balancer. Next time: Local Settings and Policy Read More...
|
-
Does WCF ever cache the DNS lookup for a service address? How do I clear this cache? There is just a tiny amount of caching that I know about. The TCP and MSMQ transports use a shared cache for recently used addresses. I think for MSMQ that this is only a factor for custom dead letter queue addresses. In any case, this cache is to speed up lookups in the case that you're hitting the same address over and over again very quickly. There's no way to clear the cache, but the lifetime of entries is two seconds so you'd have to be changing DNS settings extremely often to notice this. If you're getting stale lookups, then it's probably an operating system or server cache that is causing that. Next time: Moving Services and User Principals Read More...
|
-
My web service needs to periodically broadcast messages to clients. The service is an Internet-facing application hosted inside of IIS. What’s the best way to do this? The big limitation in this scenario is that your clients might be behind a firewall and non-addressable. There are basically two architecture camps for broadcasting messages to clients over the Internet. The push architecture camp has the clients maintain a continuous connection to the server and pushes data out at each update. The pull architecture camp has the clients periodically poll the server to see if there’s any new data. Both of these architectures are widely used and they trade latency versus resources off of each other. There are a few other architectures that work locally as well but aren’t as useful over the Internet, such as multicasting and callbacks. I’m just going to pick one of these and talk about using a push architecture. The basic way to build a push architecture is to have clients connect to the server and then the server holds the connection open indefinitely to send messages. If your service is hosted in IIS version 6 or below, then you don’t have a lot of choice about the network protocol. Pushing data from the server to client is difficult with HTTP because the protocol is built on top of the request-reply model. A typical way of using HTTP to push is to make an empty request and send back a response using the chunked transfer encoding. Chunked transfers allow an HTTP server to send the response in pieces without having to specify the total length of the response up front. Normally, the client knows the message is done when the connection is closed or the pre-announced content length is reached. Neither of those options work in this case. Instead, the server needs to define some framing protocol so that the client can tell when the individual messages are done. The easiest way to get chunked HTTP transfers in WCF is to use the HTTP transport with streaming enabled. If your service is hosted in IIS 7 (the Vista/Longhorn version of IIS), then you are able to pick other network protocols for your service, such as TCP. TCP is inherently duplex so it works across the Internet for server-initiated transfer of messages without having to connect back through a firewall. Since TCP is duplex, you can write a duplex contract that gives you a very nice programming model for sending the broadcasts. This is a lot less work on your part than the equivalent setup needed with HTTP. Read More...
|
-
I have a long-running service operation that needs to receive a response. What options do I have for designing my web services? The problem that most people run into with long-running operations is that the operation eventually hits some quota value and times out. For instance, if you want to run the operation over an HTTP connection, you have to configure the HTTP transport to not give up waiting before the operation completes. This configuration consists of setting the send and receive timeouts of the channel to some maximum bound for the operation time. Setting the timeouts correctly requires you to know what a maximum bound is for your operation. If you don’t know of a maximum bound or the expected operation time is poorly defined, then it becomes very difficult to set a reasonable quota value. Quota values that are too small cause the operation to be unnecessarily aborted. Quota values that are too large weaken your security and makes you more vulnerable to malicious attackers tying up your resources. Large operation times also mean that you’re more susceptible to dropped connections or network glitches that cause the operation to be aborted. One alternative to holding a transport channel open during the entire operation is to use an asynchronous delivery mechanism (this shouldn’t be confused with asynchronous method calls). Asynchronous delivery, such as a queue, allows you to drop messages off without having a continuously stable network connection to the other party. Reliable messaging is also a way to recover from dropped connections. However, neither of these methods helps you bound your maximum operation time for request-reply style messaging. The basic solution to unbounding your operation time is to decouple the request and response of the operation into a pair of one-way operations. A one-way operation on the service is used to receive the request and a one-way operation on the client is used to receive the response. Correlation is still needed to associate the request and response together. This correlation is no longer defined by the use of a single network connection but instead is some property associated with the message. Each one-way call has a well-bounded and predictably short operation time. The one-way model is not perfect. Transports that do not support symmetric message transfer, such as HTTP, are harder to use because you may need to address client machines and poke holes through firewalls. However, that is a tradeoff you may need Read More...
|
-
Today I spent some time reflecting on the proliferation of crummy date formats in network protocols. The particular plague was the HTTP/1.1 specification, which includes not one but two date formats that are obsolete yet required to be supported. There's only one permitted date format but everyone is supposed to be able to read RFC 850 and asctime dates just in case. Does your date parser even support asctime? Libraries have a lot of code tied up in handling these kinds of corner cases, and that's likely to contain a pile of bugs given the lack of users actually exercising these options. Support for these kinds of standards quirks slowly seems to be dropping off as newer standards appear. Perhaps one day we'll be rid of the legacy of two-digit years and times without time zones. A moment of silence generally commemorates a past event, but in this case I'm being optimistic. Read More...
|
|
|
|