Matthias Vallentin

High-Level Network Traffic Profiling

2011-12-16T08:00:00Z

The size of an event is given by the size of all its arguments (plus a little bit of meta information, e.g., creation timestamp). How do we get such meta data though? To date, Bro has limited meta programming support, which makes it difficult to obtain these details. Thus I added a new meta event to Bro, which the core generates in addition to each regular event. Here is an example that shows how to handle this event:

event meta_event(name: string, timestamp: time, size: count)
    {
    print fmt("%s\t%f\%d", name, timestamp, size);
    }

You can find a more complete example that writes information about all events to a separate file meta.log in my brospects repository. To use this meta programming functionality, you need to use the topic branch topic/matthias/meta-analysis. A fresh install might look like this:

git clone --recursive git://git.bro-ids.org/bro.git
git checkout topic/matthias/meta-analysis
./configure --prefix=PREFIX  make  make install

To give a real world example of what you can do with this functionality, let us analyze a trace captured at UC Berkeley (UCB) on October 17, 2011, at 2:35pm for 10 minutes. At UCB’s network border, a Bro cluster of 26 nodes monitors two 10 Gbps links. The full packet trace has a size of 219 GB and contains 284 million packets in 6.5 million connections. When looking at the trace sizes on the 26 nodes, we observe a median size of 8.1 GB () in a range from 6.3 to 15 GB.

Let us inspect one 7.7 GB trace of a single cluster node in more detail. The two Figures below show the number of events per second, once over time and once their empirical distribution.

Better NIDS Performance by Tracking Roaming Users

2011-06-22T07:00:00Z

Firesheep had a noticeable impact on several large Web 2.0 players: Twitter, FaceBook, GitHub, Dropbox, Slicehost, and others now offer either opt-in HTTPS or have switched to a full HTTPS deployment in order to protect their users from session hijacking attacks. In an earlier article I described how to write a Bro script that passively detects Firesheep. In this article, I discuss how tracking roaming users in DHCP-enabled networks can improve the false positive rate of network intrusion detection system (NIDS) detectors of this kind.

Passive Detection

Passive monitoring is less intrusive than active approaches, such as the Blacksheep Firefox extension which sends fake cookies and looks whether a Firesheep user tries to utilize them, or the FireShepherd command-line utility which simply floods the network with zillions of fake cookies with the goal to crash Firesheep (and thereby the browser). In general, such active “defense” mechanisms do not scale to larger networks and may have even undesirable side-effects in affecting their users. Therefore, I believe that purely passive detection mechanisms are more attractive in practice.

However, false positives are an issue with the passive sidejacking detector. If a network uses DHCP, chances are that the same user receives a different IP addresses at some point in time. But when the sidejacking detector has not yet expired the cookie value associated with the old IP address and the cookie reappears under a new IP address, a false alarm is generated. This happens due to the lack of a crisp user definition: IP address are ephemeral in DHCP networks and cannot be used to analyze user activity over time without taking roaming into account.

Reducing False Positives

How should we ? The key insight is to keep track of all IP addresses under which the same user appears, where a user is uniquely identified by its MAC address. You might think that this is obvious, so why not always use MAC addresses? For small coffeeshop networks, this would probably work just fine: a typical deployment has a single broadcast domain and the NIDS can be part of it. However, the issue is with larger networks, partitioned into several broadcast domains, where traffic from access points is aggregated over a hierarchy of switches. In such scenarios, the NIDS might not even see the MAC addresses of the users because it is deployed upstream (usually at the border or DMZ).

With Bro, it is possible to simply ship the missing layer-2 information to the machine on which the detector runs. This works in monitoring infrastructures consisting of network-wide sensors (or worker nodes) which can see and forward link-layer events to a central manager nodes that runs the scripts processing the events. Our NIDS cluster paper has more details on this architecture.

Assuming that we can see the relevant broadcast traffic, how do we learn IP-to-MAC mappings of roaming users?

Clearly, we can learn IP-to-MAC mappings by monitoring DHCP traffic. In particular, the DHCP ACKs contain the binding between the two addresses, as discussed in the previous sidejacking article.
I was pleased when Jordi Ros-Giralt from Reservoir Labs sent me a patch that incorporates another type of broadcast traffic: ARP. Similar to DHCP ACKs, ARP replies also contain the mapping from IP to MAC address.

Bro’s new roam.bro script keeps track of these mappings in two global tables:

global ip_to_mac: table[addr] of string
    &read_expire = alias_expiration &synchronized;

global mac_to_ip: table[string] of set[addr]
    &read_expire = alias_expiration &synchronized;

Event handlers for the dhcp_ack and arp_reply events populate these tables. For example, here is the DHCP ACK handler:

event DHCP::dhcp_ack(c: connection, msg: dhcp_msg, mask: addr,
    router: dhcp_router_list, lease: interval, serv_addr: addr)
{
    local ip = msg$yiaddr;
    local mac = msg$h_addr;
  
    if (ip !in ip_to_mac)
        ip_to_mac[ip] = mac;
  
    if (mac !in mac_to_ip)
        mac_to_ip[mac] = set() &mergeable;
  
    add mac_to_ip[mac][ip];
}

One note for cluster users: the &synchronized attribute makes sure that these tables have the same content across all nodes in a cluster. The &mergeable attribute ensures that the address sets of the table mac_to_ip are correctly synchronized. That is, the local sets of the worker nodes are merged at the manager. You can think of it conceptually as creating the union of all sets instead of just using the value of the last update that would override previous updates. See our NIDS cluster paper for more details about these attributes. At this point I should mention that I factored the roaming logic into a separate Bro script roam.bro so that other scripts can also make use of the provided data structures.

Jordi’s patch also includes a new type of Notice, SessionCookieRoamed, which is generated when the same MAC address reappears under a different IP address. The Figure below shows the updated decision tree of the sidejacking detector; red rectangles correspond to a Notice and round nodes represent a decision.

The latest version of the detector and the roaming script are available from the Bro scripts git repository.

Summary

This article describes how to reduce false positives of network-based detectors that base their notion of accountability on IP addresses. I show how flexible cross-layer NIDS, like Bro, can make such detectors more robust by incorporating link-layer context (i.e., MAC addresses). Although the running example is sidejacking, this improvement works in all scenarios where IP-level state is kept across TCP connections.

Omni, Take Two

2011-06-15T07:00:00Z

Did you know that 2 equals 1? Here is the proof:

(1) X = Y                         ; Given
(2) X^2 = XY                      ; Multiply both sides by X
(3) X^2-Y^2 = XY-Y^2              ; Subtract Y^2 from both sides
(4) (X+Y)(X-Y) = Y(X-Y)           ; Factor
(5) X+Y = Y                       ; Cancel out (X-Y) term
(6) 2Y = Y                        ; Substitute X for Y, by equation 1
(7) 2 = 1                         ; Divide both sides by Y
                -- "Omni", proof that 2 equals 1

If you think this is too easy, here is another neat proof spotted by my friend Avital Steinitz while teaching a Signals and Systems course.

Consider the function given by the rule , also known as the phaser with frequency . We show that this function is in fact the constant function :

A Garden Variety of Bloom Filters

2011-06-14T07:00:00Z

In this article, I explain how Bloom filters work and introduce several variations that evolved as a result of extensive academic treatment of this topic. Moreover, I introduce my own implementation of these Bloom filters, libBf, a header-only C++11 library that comes with a BSD-style licence.

Whenever you have a set or list, and space is an issue, a Bloom filter may be a useful alternative.
–Mitzenmacher

Introduction

A Bloom filter is a randomized synopsis data structure that allows for set membership queries. Its space-efficient representation comes at the cost of false positives, i.e., elements can erroneously be reported as members of the set. In practice, the huge space savings often outweigh the false positives if kept at a sufficiently low rate.

Bloom filters have received a great deal of attention not only by the research community but also in practice. For example, Google Chrome uses a Bloom filter to represent a blacklist of dangerous URLs. Each time a user is about to navigate to new page, the corresponding URL is mangled, hashed, and then compared to a local Bloom filter that represents the set of all malicious URLs. If the Bloom filter reports that the URL is in the set, the browser sends the hash of the URL to the Safebrowsing server to confirm that the URL is indeed malicious to compensate a potential false positive. That is, all checks are performed locally, but when the user surfs to a malicious URL, an extra round trip to the Safebrowsing server occurs.

Another example is the squid web proxy which uses Bloom filters to represent cache digests, which caching servers use to periodically exchange the objects they contain. There are many more examples of Bloom filter applications, for instance in peer-to-peer applications, routing protocols, IP traceback, resource location, etc. Broder and Mitzenmacher give a good survey of network applications.

Bloom Filters

Before we delve into the discussion, let us agree on some common notation.

Terminology

Universe
distinct items
independent hash functions
Vector of cells, i.e.,
Set:
- where and
Multiset / Stream:
- where and
- counters of
- multiplicity (frequency) of
Bloom filter estimate denoted by hat:
Probability of a false positive (FP):
Probability of a false negative (FN):
Capacity , i.e., is the maximum number of items a Bloom filter can hold until a given can no longer be guaranteed
A Bloom filter is full when then number of added items exceeds

Basic

Burton Bloom introduced the original Bloom filter in 1970, which we refer to as the basic Bloom filter. It uses a bit vector with and independent hash functions that map items in to the range . (Unlike in the implementation, we start at index 1 for the formal treatment.) All bits in are initialized to 0. To insert an item , we set the bits at positions in to 1. To test whether an item is a member of , we examine the bits at positions in . If any of these bits is 0 we report . Otherwise we report that , although there remains some probability that . This type of error is a false positive (FP) and also known as Bloom error . It occurs because other elements in also map to the same positions.

To compute the probability of a Bloom error, we start off with an empty bit vector and insert an item. This is the same as independently (and uniformly) choosing bits and setting them to 1. Thereafter, the probability that a certain bit in is still 0 is

Afer insertions, the probability that a certain bit is 1 is

Testing for membership involves hashing an item times. Thus the probability of a Bloom error is

For fixed parameters and , the optimal value that minimizes this probability is

For , we have hence . Moreover, for a desired FP probability we can compute the number of required bits by substituting the optimal value of :

Multisets

A basic Bloom filter can only represent a set, but neither allows for querying the multiplicities of an item, nor does it support deleting entries. We use the term counting Bloom filter to refer to variants of Bloom filters that represent multisets rather than sets. In libBf, we call a counting Bloom filter a basic Bloom filter with width parameter .

Counting

In a counting Bloom filter, inserting an item corresponds to incrementing a counter. Some variants also feature a decrement operation to remove item from a set. As soon we allow for deletions we introduce false negative (FN) errors. These occur when removing an item that itself was a FP. The probability of a FN is bounded by .

When we get the count of an item , we compute its set of counters and return the minimum value as frequency estimate . This query algorithm is also known as minimum selction (MS).

There are two main issues with counting Bloom filters:

Counter overflows
Choosing

The first problem exists when we the counter value reaches and cannot be incremented any more. The most natural thing to do is simply not continuing to count rather than overflowing and restarting at 0. However, this introduces undercounts, which we also refer to as FNs.

The second problem concerns the choice of the width parameter . If we choose very large, the space gains of a Bloom filter diminish. There will also be a lot of unused space, manifesting as unused zeros. If we choose too small, we will reach the maximum counter value to fast. Choosing the right value is a difficult trade-off that also depends on the data. It has been shown that for uniform distributions a value of works well.

Bitwise

The bitwise Bloom filter is a combination of counting Bloom filters with bit vectors , each of which have cells, hash functions, and width where . It aims at solving both of the overflow and space problem of the counting Bloom filter.

To add an item , we first look at the counters in the first level . If there is enough room (i.e., width) available, we simply perform the increment. If the counter overflows, we insert into and remove it from . In this fashion, the counter value is unbounded as we can always add more levels. However, the item has to be hashed times with a total of hash functions.

In order to read the counter of an item, we combine the binary representation of all the levels. Let be the counter value at level . Then we compute the counter value as

Spectral

The spectral Bloom filter is an optimized version of the counting Bloom filter. It consists of two extra algorithms in addition to MS and introduces a more space-efficient data structure to represent counters.

Let us review the MS algorithm. When querying an item , MS uses the minimum counter value as frequency estimate, i.e., . Cohen and Matias claim that and for all .
The second spectral algorithm is an optimization for the add operation. When adding an item to the Bloom filter, the minimum increase (MI) algorithm only increments the minimum counter value(s) . The rationale behind this is that is always the most accurate count, thus MI results in the fewest possible increment operations.

Because not all counters are incremented on inserts, the effect of deletes is significantly worse and the number of FNs becomes unbounded. Thus, the MI algorithm should not be used when removing of items from a Bloom filter. Cohen and Matias claim that and that if is drawn uniformly from , then .
The third algorithm is recurring minimum (RM) and involves two Bloom filters, and . The key insight behind RM is that items that experience Bloom errors are less likely to have recurring minima counter values. Cohen and Matias found empirically that this applies to approximately 20% of the items. Such items with a unique minimum are maintained in the second Bloom filter to reduce the discrepancy between and .

To query an item according to the RM algorithm, we look first into the first Bloom filter and check if has a recurring minimum. If so, we return the minimum counter value. Otherwise we look the minimum counter value from the second Bloom filter, unless it is 0. If it is 0 (i.e., does not exist), we return the minimum counter from the first Bloom filter.

Since all the items are inserted into the first bloom filter, the RM optimization does at least as well as the MS algorithm, yet has usually better error rates because a second filter holding fewer items is used for items which experience higher error rates.

The fancy data-structure takes space, where is the number of distinct items and . For details, please refer to the paper.

Aging

A problem all the above Bloom filter variants is that they eventually fill up over time when dealing with a large set or stream of data. This means that at some point the Bloom filter becomes unusable due to its high error rates. There exist various scenarios where one would like to “age out” old items that have been added a long time ago. For example, we might want to estimate only recent items or we have a very limited amount of space available.

Although counting Bloom filters have a delete operation, it is often impossible to retain old items in memory. Thus we do not know their counter positions in the bit vector anymore, otherwise we would simply decrement their count. What we want is a Bloom filter that has sliding window semantics, as illustrated by the Figure below.

To support a sliding window, we would like to have Bloom filter that acts like a FIFO. In the following, we discuss two different Bloom filter flavors that aim at providing this property.

Stable

The stable Bloom filter is essentially a basic Bloom filter with an underlying bit vector with a fixed cell width . However, counters do not represent the multiplicities of the items but rather their age. Thus the interface supports only set membership queries.

To insert an item, we decrement cells chosen uniformly at random. Thereafter, we set the counters of all cells to their maximum value of .

Deng and Rafiei have shown that the fraction of zeros will eventually become constant. When having reached this stable point, the probability of a Bloom error is

A²

The Bloom filter, also known as active-active buffering, provides another type of FIFO. It uses two single-bit vectors and where . Unlike the spectral RM algorithm, one Bloom filter is not a subset of the other, so an item can be in either Bloom filter.

To query for an item, we return true if exists in either or . To insert an item , we simply return if it already exists in . Otherwise we insert it in and test whether has reached its capacity. If it is full, we flush and swap and . Thereafter we insert the item in (the old ).

One advantage of the Bloom filter is space-efficiency, since one bit vector is always full. Let the subscript denote the value of the Bloom filter. The probability of a Bloom error is

and the optimal value for and are:

libBf

As part of a class project for the course Combinatorial Algorithms and Data Structures in Spring 2011 at UC Berkeley, I decided to write a C++11 library of Bloom filters, libBf, which implements the above discussed Bloom filters. Slides of the final presentation are also available; they go a little deeper into the algorithmic details.

I only presented a few Bloom filter types in this article, but active research in this field yielded many more variations. For example, the dynamic and scalable Bloom filter are two variants that grow dynamically as soon as more items are added. Bloom filters can also be compressed, e.g., when sending them over the network. Distance-sensitive Bloom filters give more than a binary answer or count when asking for an item: they also return if an item is close to another item in the set. Finally, there exist Bloomier filters which extend the set membership query model and counting notion to computations of arbitrary functions.

Cuckoo hashing is a related space-efficient alternative to Bloom filters. Moreover, Adam Langley has some interesting thoughts on Golomb Compressed Sets.

Analyzing Facebook Webchat Sessions with Bro

2011-06-12T07:00:00Z

The Facebook webchat allows you to chat with your friends while having a Facebook window open in the browser. In this post, I describe how the webchat protocol works and show how to write a Bro script that analyzes chat sessions.

The Facebook Webchat Protocol

As a traffic analyst, you might wonder how one can get insight into the webchat protocol details and how to work with it at a higher level. After all, who wants to write boilerplate code whose only purpose is to fight the representation of the data rather than analyzing the data itself? Let us extract messages from a Facebook webchat conversation and put them into Bro data structures where they are easy to manipulate, print, and react upon. This involves parsing the JSON objects, which look like this:

for (;;);{"t":"msg","c":"p_100002331422524","s":4,\
  "ms":[{"window_id":1985081376,"type":"unfocus_chat"}]}

The above example is an unfocus_chat event sent over the AJAX channel, indicating that the user placed the focus somewhere else on the page, away from the chat window. Here is another HTTP body:

for (;;);{"t":"msg","c":"p_100002331422524","s":6,"ms":[{"msg":{
  "text":"So I need the URL, dude.  What is it?","time":1303218454567,\
  "clientTime":1303218453582,"msgID":"2755876075"},"from":100002331422524,\
  "to":100002297942500,"from_name":"Mondo Cheeze","from_first_name":"Mondo",\
  "from_gender":2,"to_name":"Udder Kaos","to_first_name":"Udder",\
  "to_gender":2,"type":"msg"
}]}

This one is an actual chat message. The nice thing is that such messages are self-contained and include quite some meta information: contents, timestamps, names, unique IDs, and even genders.

The Bro Script

Alas, the HTTP body is a big fat opaque string with no structure and parsing nested data in strings with just regular expression is clunky at best. (For those who have spare cycles: an extremely useful project would be to create a Bro analyzer that exposes first-class types of the DOM tree of a document and script-level primitives for basic JavaScript analysis.) A crude way to do this is splitting the string on ,", then finding the right key-value pairs, and populating the following Bro data structures with the parsed data:

type chat_message: record
{
    timestamp: string;  # Message timestamp.
    from: string;       # Name of the sender
    to: string;         # Name of the recipient.
    text: string;       # The actual message.
};

type chat_session: record
{
    start: time;        # Unix timestamp of first message.
    end: time;          # Unix timestamp of last message.
    n: count;           # Total number of messages in session.
};

At this point it is easy to generate NOTICES with chat messages, look for suspicious messages, and more generally, leverage Bro’s full scripting language more effectively. I wrote a basic script that uses these data types to dump a chat session between two buddies. You can download it from my brospects repository, which contains a mixed bag of experimental Bro scripts that are either in preparation for or don’t make it into the official Bro scripts repository. Here is the output of facebook.bro of a sample Facebook webchat session:

1303218454567 (Mondo Cheeze - Udder Kaos) So I need the URL, dude.  What is it?
1303218465938 (Udder Kaos - Mondo Cheeze) the URL?
1303218474259 (Mondo Cheeze - Udder Kaos) Yeah for the secret image
1303218481721 (Udder Kaos - Mondo Cheeze) ok lemme see
1303218495626 (Mondo Cheeze - Udder Kaos) Someone could be sniffing this conversation, be sure to send it safely
1303218503972 (Udder Kaos - Mondo Cheeze) ?
1303218570782 (Mondo Cheeze - Udder Kaos) Cmon we talked about this.  Encrypt it with WonderCipher-92 \
                                           and send me the Base64 encoding of the hex.  Usual key.
1303218587568 (Udder Kaos - Mondo Cheeze) 'k.  So here it is:
1303218595067 (Udder Kaos - Mondo Cheeze) NmQwMDJjZDdhZTdlYmYxNTc5MGVjZDc1YTYxNDk1OGE0ZTRhYjAzOTVi
1303218618252 (Mondo Cheeze - Udder Kaos) What's the IV
1303218624712 (Udder Kaos - Mondo Cheeze) huh?
1303218637197 (Mondo Cheeze - Udder Kaos) Initialization vector, you maroon.  WC-92 is a stream cipher, you know
1303218667601 (Udder Kaos - Mondo Cheeze) oh yeah.  I used my birthday, all as one number.
1303218685436 (Udder Kaos - Mondo Cheeze) you *do* remember it, right?
1303218700515 (Mondo Cheeze - Udder Kaos) yeah your an April Fool, not hard to remember
1303218710402 (Udder Kaos - Mondo Cheeze) heh
1303218718486 (Mondo Cheeze - Udder Kaos) K gimme a sec to decrypt then.
1303218733463 (Mondo Cheeze - Udder Kaos) Hey idiot this isn't the secret, it's Google's home page.
1303218745028 (Udder Kaos - Mondo Cheeze) whoops hang on, blew my cutpaste
1303218767633 (Udder Kaos - Mondo Cheeze) okay, here's the right one:
1303218776922 (Udder Kaos - Mondo Cheeze) NmQwMDJjZDdhZTdlYmYwMDY3MGRjZDdlYjA1NDlhODQ0ZjA1YmEyNDRm
1303218800303 (Mondo Cheeze - Udder Kaos) And?
1303218807537 (Udder Kaos - Mondo Cheeze) and what
1303218815022 (Mondo Cheeze - Udder Kaos) What's the IV
1303218824330 (Udder Kaos - Mondo Cheeze) huh?
1303218839537 (Mondo Cheeze - Udder Kaos) yo maroon same thing as we just discussed a moment ago, sheesh
1303218855518 (Udder Kaos - Mondo Cheeze) oh that yeah like I said my birthday
1303218869728 (Mondo Cheeze - Udder Kaos) You used the same IV as before????
1303218889893 (Udder Kaos - Mondo Cheeze) right, otherwise how would I remember it?
1303218900257 (Mondo Cheeze - Udder Kaos) YOU BOZO

In summary, this is an example of how to translate low-level representation of communication into higher abstractions that are easier to work with. After creating first-class Bro types for the involved entities, one can now leverage the real power of Bro’s scripting language.

Taming the Sheep: Detecting Sidejacking with Bro

2010-10-29T07:00:00Z

Update June 22, 2011 You might also want to read the follow-up article about reducing false positives by incorporating ARP traffic.

The Firefox extension Firesheep brings sidejacking (or cookie hijacking) to the masses. Its convenient point-and-click interface allows even unskilled users to hijack their friend’s Facebook and Twitter accounts from a public wireless network, such as a coffee shop or at the airport.

This is a relevant issue because most popular Web 2.0 services only use TLS for the initial authentication and then drop back to unencrypted HTTP throughout the rest of communication. As a consequence, the session cookies fly around in clear text and are easily captured.

Detection

As network operators and security analysts, we would like to know when session hijacking occurs. In the following, I present the design and implementation of a sidejacking detector that I wrote for the Bro intrusion detection system.

Design Considerations

The detection strategy depends on the network topology and available context. One problem is how to define a user. In flat IP address space, it makes sense to identify a user by its IP address, but this would fail in a NAT environment, where a single IP accomodates several users. For NAT scenarios, one could define a user as a pair of IP address and User-Agent header. For large NATs or NATs with identically configured machines, this latter notion of user could be prone to false positivies, but this trade-off has to be faced. Therefore, the notion of what constitutes a user can be configured in the detector.

Another issue poses the presence of absence of DHCP context. Consider a hotspot or coffeeshop network operator with a private IP space. If Bro can see DHCP traffic, we can crisply identify a user by its MAC address and do not have to resort to high-level notions, such as provided by HTTP headers. In a static setup, however, this context will likely not be available. At the same time, if a different address reuses a session cookie in a static IP topology, we probably observed a sidejacking attack. Furthermore, if the same address uses the same sesssion cookie from a different user agent, we report such activity.

There is another subtlety: the cookie header consists of a list of key-value pairs, yet only a subset of those represent a user session while the others could be random. Simply comparing the entire cookie string against a value seen in the past would thus be prone to false negatives. Hence we have to restrict ourselves to the relevant session-identifying subset. In fact, this is how Firesheeps works: it ships with site-specific handlers that define the cookie keys to extract and then sets only those to impersonate a user. This motivates the following design: if a specification for a particular service is available, restrict the cookie to the relevant fields, and otherwise use the cookie as a whole. The default set of known services that ships with the detector is based on all handlers Firesheep currently implements.

Decision Trees

Given these considerations, how should the detector operate? Clearly, we need a flexible mechanism that can be configured to meet the specific conditions. Let us distinguish the two notions of users. If we have the opaque NAT setting, the detector operats according to the Figure below.

Updating the cookie context is not really an actionable item but means that the per-cookie state (such as when it was last seen) is being updated. In comparsion, the decision tree for a flat address space looks a little more elaborate.

Note that session cookie reuse is only reported when the user is defined as IP address.

Operation

Simply using the sidejacking detector is straight-forward and does not require detailed knowledge of Bro. Since it is implemented as a Bro script, running it only requires passing the filename of the script on the command line:

 bro -i eth0 sidejack

When Bro detects session hijacking, it will report a Sidejacking notice:

TIMESTAMP Sidejacking ATTACKER ADDRESS (ATTACKER HTTP SESSION ID) @ 'ATTACKER USER_AGENT' hijacked SERVICE session (VICTIM HTTP SESSION) of VICTIM ADDRESS @ 'VICTIM USER AGENT' last seen at SESSION COOKIE LAST SEEN via cookie SESSION COOKIE

Similarly, session cookie reuse is reported as SessionCookieReuse and has the following format:

ATTACKER ADDRESS (ATTACKER HTTP SESSION ID) reused SERVICE session VICTIM SESSION ID in user agent 'ATTACKER USER AGENT', where previous user agent was 'VICTIM USER AGENT' and last seen at SESSION COOKIE LAST SEEN via cookie SESSION COOKIE

Extending the detector with a new site profile is also not too complicated. I would recommend to write a new Bro script and leaves the original one unmodified. Similar to writing a new Firesheep handler, a corresponding detector for a fictive service Foo at foo.com where a user session comprises the two cookies session_id and auth_token would look as follows:

@load sidejack

redef HTTP::cookie_list +=
{
    ["Foo"] = [$url=/foo.com/, $keys=set("session_id", "auth_token")]
};

If the cookie keys are dynamically generated, as it is the case with Wordpress, one can also specify a regular expression instead:

redef HTTP::cookie_list +=
{
    ["Foo"] = [$url=/foo.com/, $pat=/session_[0-9A-F]+/]
};

Telling the detector to use DHCP involves mereley flipping a switch:

redef HTTP::use_dhcp_aliases = T;

Similarly, changing the notion of a user to include the user agent:

redef HTTP::user_is_ip = F;

To extend the analysis to all cookies and not only the listed known service profiles:

redef HTTP::known_services_only = F;

Finally, it is possible to tweak the cookie expiration interval, i.e., the time how long Bro remembers a cookie session. The example below tells Bro to keep cookies for 10 minutes only.

redef HTTP::cookie_expiration = 10 min;

If an attacker uses the cookie after the expiration interval, Bro will not detect it. Naturally, there is a trade-off between accumulating state in memory and opportunity for evasion.

Mitigation

By weaponizing the general public, Firesheep will hopefully push vendors to switch TLS-only deployments more rapidly. During this painful migration period, techniques like HTTP Strict Transport Security (HSTS) could prove useful. A HSTS-enabled server sends a Strict-Transport-Security header to the client that specifies a time period during which the client should use encryption only. That is, the server upgrades the connection if the client tries to use unencrypted HTTP.

Bro Code

Practical security is all about raising the bar. Hopefully this detector allows network operators to pinpoint the use of sidejacking attacks, such as facilitated by Firesheep, and take accountable action. The detector is part of Bro script repository which we created as an official place for user-contributed scripts. The latest version of the script can be downloaded here.

Probability and Statistics Cheat Sheet

2010-10-07T07:00:00Z

Update June 5, 2011 The probability and statistics cheat sheet now has it’s own page. Furthermore, I decided to rename it to cookbook because the sheer number of pages stretches the definition of cheat sheet quite a bit.

While I started to explore the many univariate probability distribution relationships last year, I created a PDF to help me remember them. This document quickly grew into a basic probability theory reference and its substantial parts evolved while I was taking the excellent course STAT 200B at UC Berkeley taught by Cari Kaufman. Today, it is more like a cookbook and helps me to swap back in the relevant context when facing statistical challenges.

The purpose of this cookbook is to bundle essential statistical knowledge with a unified notation. If you find any mistakes or have suggestions for further topics, I’d appreciate if you contact me. I will continue to add material that I deem useful.

Download

Screenshots

Should You Trust Your SSL Certificate?

2010-03-30T07:00:00Z

Is your SSL secure? I’m not sure that the answer is quite so binary after reading the paper Certified Lies: Detecting and Defeating Government Interception Attacks Against SSL by Christopher Soghoian and Side Stamm. In a so-called compelled certificate creation attack, a government agency forces a CA to issue a fake certificate to intercept an SSL-encrypted (or more precisly: TLS-encrypted) session without triggering a warning in the victim’s browser. This works seamlessly, because it does not matter who issued a specific certificate as long as the browser sees a valid chain of trust terminated with a known root certificate. Although the authors claim that this attack is exercised in practice, data-driven evidence is yet lacking.

Proactive Protection

In order to assess whether or not one’s current TLS session is being intercepted, the authors have developed a Firefox extension called CertLock, which extends the browser history by collecting additional certificate information such as its hash, the name and country of the issuing CA and the website, and the trust chain up to the root CA. Each time the user revisits a TLS-protected site for which a certificate in the history exists, CertLock compares their two hash values. If a mismatch is detected, CertLock next compares the issuing CAs’ country. In the event that they differ, CertLock presents a warning to the user; otherwise the page loads without any warnings.

Although CertLock alleviates some problems with the compelled certificate creation attack, it is still limited in the following ways.

Trust-On-First-Use

When visiting a TLS-protected website for the first time, the user immediately faces a dilemma: how to trust the certificate chain when there is nothing to compare to. Because CertLock operates by comparing the current certificate with one from the history, it cannot detect if the first encountered certificate chain is authentic. That is, CertLock can only protect the user for sites that have already been visited in the past and deemed secure.

When planning a trip to a potentially hostile environment (e.g., DEF CON or China), a common recommendation is to use a fresh laptop, or at least to replace the laptop’s harddrive and start over with a fresh system installation. This is precisly the scenario where CertLock would be necessary, but cannot function due to the lack of browsing history.

Ground Truth

A well-known issue with anomaly detection is the presence of an adversary during the training phase, who can later conduct unnoticed attacks when the system is in live operation. CertLock cannot distinguish counterfeit from real certificate trust chains when trained maliciously. In fact, the opposite of the intended behavior may occur: a benign certificate may be misclassified as untrustworthy, and a forged certificate may be blindly accepted. This problem highlights the need for a trustworthy past certificate history, otherwise it is impossible to make an accurate decision.

False Negatives

CertLock suffers from false negatives when (i) the actual and compelled CAs are from the same country and (ii) the certificate differs from the one in the history but the issuing CA has not changed. At the same time, the number of false positives is greatly reduced this way, which is vital to get the user’s attention in this scenario.

User Failure

When I asked two of my friends what their impression was of CertLock’s displayed warning, I was surprised to hear “it looks like spam - what does Russia have to do with the site I am visiting?” Granted, the authors acknowledge the room for improvement on the user interface, and their intention to brush up the warning’s design. However, given the population’s limited geographic understanding of international relations, this may be a challenging task.

Retrospective Analysis

In addition to the aspects above which highlight the need for better proactive defenses, opportunities for forensic analysis are equally important to appreciate. Network traffic traces may contain evidence of the compelled certificate creation attack. Bro already supports extraction of certificates which minimizes the programmatic efforts required to implement a detector. To address the problem of ground truth, the traces should ideally span long time frames and be recorded from multiple vantage points. In the future, I plan to write a Bro script to detect this type of TLS tampering, which could be particularly useful to unveil targeted attacks, as these tend to live under the radar of the standard network intrusion detection system.

Update March 3, 2011 It is great to see some momentum in the community regarding this topic. Erik Hjelmvik blogs about the forensic analysis of TLS certificates to detect man-in-the-middle attacks. To extract certifcate files, he suggests NetworkMiner, a GUI tool for Windows.

Well, Bro’s TLS analyzer also provides this basic extraction functionality, yet more beyond that. For example, here’s a subset of events the TLS analyzer generates:

ssl_conn_weak(name: string, c: connection)
ssl_certificate_seen(c: connection, is_server: bool)
ssl_certificate(c: connection, cert: X509, is_server: bool)
ssl_conn_attempt(c: connection, version: count, ciphers: cipher_suites_list)
ssl_conn_server_reply(c: connection, version: count, ciphers: cipher_suites_list)
ssl_conn_established(c: connection, version: count, cipher_suite: count)
process_X509_extensions(c: connection, ex: X509_extension)
ssl_session_insertion(c: connection, id: SSL_sessionID)
ssl_conn_reused(c: connection, session_id: SSL_sessionID)
ssl_X509_error(c: connection, err: int, err_string: string)

Each time Bro raises one of these events, user-defined handler code executes. This allows for a very fine-grained analysis of TLS traffic and certificates. Such a high-level abstraction of activity exists for a numerous network protocols, which makes Bro “the Python/Ruby” for traffic analysis. That said, there are plenty of domain-specific primitives available in Bro to effectively write a detector for the compelled certificate attack.

Email Attachment Processing with Bro

2009-08-16T07:00:00Z

Malware that spreads via email is nothing new. Particularly targeted attacks against politically sensitive institutions or individuals consist of well socially engineered mails and often ship with custom 0-day malware in the form of email attachments. In order to extract such malicious attachments, I wrote a Bro policy script which records suspicious attachments to disk for later analysis. A possible application scenario would be to scan office documents for malicious JavaScript or executables for viruses. Another option would be hashing the attachment directly in Bro and comparing it against a publicly available registry, such as Seth Hall illustrates for HTTP traffic.

The script to extract attachments works by registering a callback handler for the Content-Type header in an SMTP session. Then both MIME type and the name of the attachment is examined. If either looks suspicious, Bro generates a SensitiveMIMEType or SensitiveExtension NOTICE. The user can customize the the analyzer behavior in many ways. To change the directory where the attachments are stored on disk, one can redefine the attachment_dir variable:

redef Email::attachment_dir = "foo";

The script stores the attachments by default, but this behavior can easily changed via:

# Whether attachments with sensitive MIME types should be stored.
redef Email::store_sensitive_mime_types = F;

# Whether attachments with sensitive file extensions should be stored.
redef Email::store_sensitive_extensions = F;

It is also possible to restrict or extend the regular expression used to determine whether an attachment is sensitive or not:

# Deem only application\/octet-stream as suspicious.
redef Email::sensitive_mime_types = /application\/octet-stream/;

# Restrict sensitive extensions to office documents and executables.
redef Email::sensitive_extensions =
    /[pP][dD][fF]$/
  | /[dD][oO][cC][xX]?$/
  | /[xX][lL][sS]$/
  | /[pP][pP][sStT]$/
  | /[eE][xX][eE]$/
  | /[cC][oO][mM]$/
  | /[bB][aA][tT]$/;

The script generates a file of the form ID-filename where ID is a unique attachment ID that is monotonically increasing and filename is the name of the attachment or just the MIME type if the attachment does not have a name.

The script is part of the Bro scripts git repository where you can always download the most recent version.

The Doom of Client-Side Wireless Network Security

2008-05-10T07:00:00Z

HD Moore recently announced the integration of the KARMA tools with the metasploit framework. The implications of this fusion are devastating. In an interview with Patrick Gray, HD presents the new powerful capabilities that take client-side wireless exploitation to a new level. Technically, HD rewrote parts of the original KARMA driver, included some patches, and integrated the KARMA user-land daemons into the metasploit framework.

To illustrate the new potent features of metasploit, consider the following scenario. A user opens his laptop on the plane to watch a DVD. If he ever connected to an insecure access point, it will be in his list of list of preferred wireless networks. Since the operating system attempts to connect to all known wireless networks at boot time or when waking up from hibernation, it sends out probes to look for known networks. An attacker, a couple of rows behind, responds to the probes, provides an IP address to victim by DHCP and is now rigged up to launch a multitude of client-side attacks.

Unaware of being owned, the victim’s mail client periodically tries to re-send emails laying around in the outbox. The DNS request for the SMTP server is intercepted by the attacker who returns his own address. Further, he mimics the entire SMTP connection handshake when the victim connects. Thus the victim sends his emails directly to the attacker through a fake SMTP channel. This scenario extends of course to any other plain-text protocol (HTTP, FTP, POP3, etc.). Clearly, the dominant position of the attacker yields ample opportunity for more sophisticated client-side wireless attacks, as the next examples by HD show.

Massive cookie stealing. Traditional cookie stealing presupposes that the victim actively transmits a cookie from a particular web site in order to be captured by the attacker. In contrast, this attack only requires a single HTTP request to originate from the victim to hijack all cookies from the victim’s browser. In general, only the requested site is allowed to read that particular cookie. With a malicous server responding to all client request, the attacker can bypass this restriction. When a victim sends a HTTP request, the attacker returns a chosen list of web sites (say the current top 500 sites) and the browser then tries to connect to each site with the corresponding cookie. Because all sites resolve back to the same attacker’s hostname, all cookies arrive in the hands of the attacker. Thus, by merely trying to access an arbitrary page in the Internet, the victim exposed all his cookies that correspond to entry in the attacker’s list of sites.
Browser credential theft. The next interesting step from the attacker’s perspective is it to hunt for usernames and passwords. To this end, HD wrote a little script storing all form information of the top 500 websites, e.g. forms asking for personal data, SSN, bank account number, MySpace and Facebook logins, and so on. When the victim visits any arbitrary website, the attackers returns a page full of frames that open the pages from the list and contain the saved form snippets. If the victim enabled automatic form fill-out in his browser preferences, the forms are auto-populated with sensitive user data. In addition to the form snippets the attacker delivers a malicious piece of JavaScript that grabs the form contents after filled out by the browser and sends them back to the attacker. Hence a single page visit results in a complete compromise of the victim’s login credentials and personal data.
Web-based SMB relay exploitation. Worse, if the victim happens to use Internet Explorer, a weakness in Microsoft’s SMB file sharing authentication protocol can be exploited to own the victim’s machine completely. By including a link pointing to a network file share, the victim is forced to authenticate to the attacker’s fake SMB server. This exposes the challenge key that can in turn fed back to the client. Essentially, the victim now authenticates against himself. Once connected, the incoming connection is disconnected and the new session serves as a vehicle to execute arbitrary shellcode.

Who knows what HD’s new toy features beyond the sketched scenarios? In any case, these attack vectors witness how broken the actual model of wireless security on the client-side is. While the industry tries to fix wireless encryption schemes, the actual targets, the users themselves, are not considered in the equation. These new techniques essentially render networking in any wireless environment tremendously insecure.

Writing a Linux Kernel Driver for an Unknown USB Device

2007-04-02T07:00:00Z

This article explains the creation process of a Linux kernel device driver for an undocumented USB device. After having reverse-engineered the USB communication protocol, I present the architecture of the USB device driver. In addition to the kernel driver I introduce a simple user-space tool that can be used to control the device. Although I have to delve into the specifics of a particular device, the process can be applied to other USB devices as well.

Introduction

To facilitate USB programming, the USB interface is accessible from user-space with libusb, a programming API concealing low-level kernel interaction. The proper way to write a device driver for the missile launcher would hence be to leverage this API and ignore any kernel specifics. Nevertheless, I wanted to get involved with kernel programming and decided thus to write a kernel module despite the increased complexity and higher effort.

The remainder of this article is structured as follows. After pointing to some related work, I give a quick USB overview. Thereafter, I present the reverse-engineering process to gather the unknown USB commands steering the missile launcher. To come up with a full-featured kernel device driver, I describe the kernel module architecture which incorporates the derived control commands. Finally, I demonstrate a simple tool in user-space that makes use of the driver.

Apparently I have not been the only one who played with this gadget. However, none of the existing approaches I have encountered pursue the creation of a Linux device driver for the kernel. The Launcher Library provides a user-space library based on libusb. AHmissile is a GTK+ control tool; a ncurses application is available, too. Apple users get happy with the USB missile launcher NZ project. Moreover, the python implementation pymissile supports a missile launcher of a different manufacturer. The author combined the missile launcher with a webcam in order to to create an automated sentry guard reacting on motion. I will return to these funky ideas later.

USB primer

The universal serial bus (USB) connects a host computer with numerous peripheral devices. It was designed to unify a wide range of slow and old buses (parallel, serial, and keyboard connections) into a single bus type. It is topologically not constructed as a bus, but rather as a tree of several point-to-point links. The USB host controller periodically polls each device if it has data to send. With this design, no device can send before it has not been asked to do so, resulting in a plug-and-play-friendly architecture.

Control
Interrupt
Bulk
Isochronous

Control endpoints are generally used to control the USB device asynchronously, i.e. sending commands to it or retrieving status information about it. Every device possesses a control “endpoint 0” which is used by the USB core to initialize the device. Interrupt endpoints occur periodically and transfer small fixed-size data portions every time when the USB host asks the device. They are commonly used by mice and keyboards as primary transport method. As bulk and isochronous endpoints are not relevant for our missile launcher, I skip their discussion. An excellent introduction from a programming perspective gives the Linux Device Drivers book. Below is some output from lsusb -v providing detailed information about the missile launcher.

Bus 005 Device 004: ID 1941:8021
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x1941
  idProduct          0x8021
  bcdDevice            1.00
  iManufacturer           0
  iProduct                0
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0xa0
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Devices
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.00
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      52
         Report Descriptors:
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10

The output is structured and indented like a typical USB device. First, vendor and product ID uniquely identify this USB gadget. These IDs are used by the USB core to decide which driver to give a device to. Moreover, hotplug scripts can decide which driver to load when a particular device is plugged in. Next, we can read off the maximum power usage (100 mA) in the configuration section. The subordinate interface contains apparently one interrupt IN endpoint (besides the control endpoint 0) that can be accessed at address 0x81. Because it is an IN endpoint, it returns status information from the device. To handle the incoming data we first need to understand the missile launcher control protocol.

Reverse-engenireering the USB protocol

The first step involves reverse-engineering (or “snooping”) the USB communication protocol spoken by the binary Windows driver. One approach would be to consign the device in a VMware and capture the exchanged data on the host system. But since several tools to analyze USB traffic already exist, the easier solution is to rely on one of those. The most popular free application appears to be SnoopyPro. Surprisingly I do not have Windows box at hand, so I had to install the binary driver together with SnoopyPro in a VMware.

In order to capture all relevant USB data and intercept all device control commands, the missile launcher has to perform every possible action while being monitored: moving the two axes alone and together, shooting, and moving to the limiting axes boundaries (which will trigger a notification that the axes cannot be moved further in one direction). While analyzing the SnoopyPro dump, one can easily discover the control commands sent to the missile launcher. As an example, the Figure below shows an 8 byte transfer buffer. When moving the missile launcher to the right, the buffer holds 0x00000008. Moving the launcher up changes the buffer contents to 0x00000001. It is apparently very easy to deduce the control bytes used to control the missile launcher. Unless a “stop” command (0x00000000) is sent to the device, it keeps the state of the last command. This means if the “down” command is issued, the device continues to turn until it receives a new command. If it is not possible to move further, the motor keeps up running and the gears crack with a unbearable painful sound. Upon closer examination, the interrupt IN endpoint buffer varies depending on the current device position. Whensoever an axis reaches its boundary (and creates the maddening sound), the device detects it and changes the interrupt buffer contents accordingly. This means of notification can be leveraged by the kernel developer to implement a boundary checking mechanism sending a stop command as soon as the missile launcher runs against a wall.

Here is an excerpt of the driver source showing the complete list of control commands that can be sent to the device.

#define ML_STOP         0x00
#define ML_UP           0x01
#define ML_DOWN         0x02
#define ML_LEFT         0x04
#define ML_RIGHT        0x08
#define ML_UP_LEFT      (ML_UP | ML_LEFT)
#define ML_DOWN_LEFT    (ML_DOWN | ML_LEFT)
#define ML_UP_RIGHT     (ML_UP | ML_RIGHT)
#define ML_DOWN_RIGHT   (ML_DOWN | ML_RIGHT)
#define ML_FIRE         0x10

The following bytes appear in the buffer of the interrupt IN endpoint (shown as comment) and indicate that a boundary has been reached.

#define ML_MAX_UP       0x80        /* 80 00 00 00 00 00 00 00 */
#define ML_MAX_DOWN     0x40        /* 40 00 00 00 00 00 00 00 */
#define ML_MAX_LEFT     0x04        /* 00 04 00 00 00 00 00 00 */
#define ML_MAX_RIGHT    0x08        /* 00 08 00 00 00 00 00 00 */

With all required control information in place, let’s now adopt the programmer’s perspective and delve into the land of kernel programming.

The device driver

Writing code for the kernel is an art by itself and I will only touch the tip of the iceberg. To get a deeper understanding I recommend the books Linux Device Drivers and Understanding the Linux Kernel.

As for many other disciplines the separation of mechanism and policy is a fundamental paradigm a programmer should follow. The mechanism provides the capabilities whereas the policy expresses rules how to use those capabilities. Different environments generally access the hardware in different ways. It is hence imperative to write policy-neutral code: a driver should make the hardware available without imposing constraints.

A nice feature of Linux is the ability to dynamically link object code to the running kernel. That piece of object code is called a kernel module. Linux distinguishes between three basic device types that a module can implement:

Character devices
Block devices
Network interfaces

A Character (char) device transfers a stream of bytes from and to the user process. The module therefore implements system calls such as open, close, read, write and ioctl. A char device looks like a file, except that file is “seekable” and most devices operate sequentially. Examples for char devices are the text console (/dev/console) and serial ports (/dev/ttyS0). Most simple hardware devices are driven by char drivers. Discussing block devices and network interfaces goes beyond the scope of this article, please refer to the specified literature for details.

Besides this classification, other orthogonal ways exist. As an example, USB devices are implemented as USB modules but can show up as char devices (like our missile launcher), block devices (USB sticks, say), or network interfaces (a USB Ethernet interface). Let us now look at the rough structure of a USB kernel module and then turn to particularities of the missile launcher.

struct usb_ml {
    /* One structure for each connected device */
};

static struct usb_device_id ml_table [] = {
    { USB_DEVICE(ML_VENDOR_ID, ML_PRODUCT_ID) },
    { }
};

static int ml_open(struct inode *inode, struct file *file)
{
    /* open syscall */
}
static int ml_release(struct inode *inode, struct file *file)
{
    /* close syscall */
}

static ssize_t ml_write(struct file *file, const char __user *user_buf, size_t
        count, loff_t *ppos);
{
    /* write syscall */
}

static struct file_operations ml_fops = {
    .owner =    THIS_MODULE,
    .write =    ml_write,
    .open =     ml_open,
    .release =  ml_release,
};

static int ml_probe(struct usb_interface *interface, const struct usb_device_id
        *id)
{
    /* called when a USB device is connected to the computer. */
}

static void ml_disconnect(struct usb_interface *interface)
{
    /* called when unplugging a USB device. */
}

static struct usb_driver ml_driver = {
    .name = "missile_launcher",
    .id_table = ml_table,
    .probe = ml_probe,
    .disconnect = ml_disconnect,
};

static int __init usb_ml_init(void)
{
    /* called on module loading */
}

static void __exit usb_ml_exit(void)
{
    /* called on module unloading */
}

module_init(usb_ml_init);
module_exit(usb_ml_exit);

Apart from some global variables, helper functions, and interrupt handlers, this is already the entire kernel module! But let’s start off step by step. The USB driver is represented by a struct usb_driver containing some function callbacks and variables identifying the USB driver. When the module is loaded via the insmod program, the __init usb_ml_init(void) function is executed which registers the driver with the USB subsystem. When the module is unloaded, __exit usb_ml_exit(void) is called which deregisters the driver from the USB subsystem. The __init and __exit tokens indicate that these functions are only called at initialization and exit time. Having loaded the module, the probe and disconnect function callbacks are set up. In the probe function callback, which is called when the device is being plugged in, the driver initializes any local data structures used to manage the USB device. For example, it allocates memory for the struct usb_ml which contains run-time status information about the connected device. Here is an excerpt from the beginning of the function:

static int ml_probe(struct usb_interface *interface,
                    const struct usb_device_id *id)
{
    struct usb_device *udev = interface_to_usbdev(interface);
    struct usb_ml *dev = NULL;
    struct usb_host_interface *iface_desc;
    struct usb_endpoint_descriptor *endpoint;
    int i, int_end_size;
    int retval = -ENODEV;

    if (! udev)
    {
        DBG_ERR("udev is NULL");
        goto exit;
    }

    dev = kzalloc(sizeof(struct usb_ml), GFP_KERNEL);
    if (! dev)
    {
        DBG_ERR("cannot allocate memory for struct usb_ml");
        retval = -ENOMEM;
        goto exit;
    }

    dev-command = ML_STOP;

    init_MUTEX(dev-sem);
    spin_lock_init(dev-cmd_spinlock);

    dev-udev = udev;
    dev-interface = interface;
    iface_desc = interface-cur_altsetting;

    /* Set up interrupt endpoint information. */
    for (i = 0; i  iface_desc-desc.bNumEndpoints; ++i)
    {
        endpoint = iface_desc-endpoint[i].desc;

        if (((endpoint-bEndpointAddress  USB_ENDPOINT_DIR_MASK) == USB_DIR_IN)
                 ((endpoint-bmAttributes  USB_ENDPOINT_XFERTYPE_MASK) ==
                    USB_ENDPOINT_XFER_INT))
            dev-int_in_endpoint = endpoint;

    }
    if (! dev-int_in_endpoint)
    {
        DBG_ERR("could not find interrupt in endpoint");
        goto error;
    }

    /* ... */

    /* We can register the device now, as it is ready. */
    retval = usb_register_dev(interface, ml_class);

    /* ... */
}

You might have noted the use of goto statements in this code snippet. While goto statements are generally considered harmful, kernel programmers, however, employ goto statements to bundle error handling at a central place, eliminating complex, highly-indented logic. The probe function allocates memory for the internal device structure, initializes semaphores and spin-locks, and sets up endpoint information. Somewhat later in the function, the device is being registered. The device is now ready to be accessed from user space via system calls. I will discuss the simple user-space tool accessing the missile launcher shortly. Yet before that, I present the communication primitives used to send data to the device.

The Linux USB implementation uses a USB request block (URB) as “data carrier” to communicate with USB devices. URBs are like data messages that are sent asynchronously from and to endpoints. Remember that the USB standard includes four types of endpoints. Likewise, four different types of URBs exist, namely control, interrupt, bulk, and isochronous URBs. Once an URB has been allocated and initialized by the driver, it is be submitted to the USB core which forwards it to the device. If the URB was successfully delivered to the USB core, a completion handler is executed. Then the USB core returns control to the device driver.

As our missile launcher features two endpoints (endpoint 0 and the interrupt endpoint), we have to deal with both control and interrupt URBs. The reverse-engineered commands are basically packed into an control URB and then sent out to the device. Also, we continuously receive status information from the periodic interrupt URBs. For example, to send simple data to the missile launcher, the function usb_control_msg is used:

memset(buf, 0, sizeof(buf));
buf[0] = cmd;

/* The interrupt-in-endpoint handler also modifies dev-command. */
spin_lock(dev-cmd_spinlock);
dev-command = cmd;
spin_unlock(dev-cmd_spinlock);

retval = usb_control_msg(dev-udev,
        usb_sndctrlpipe(dev-udev, 0),
        ML_CTRL_REQUEST,
        ML_CTRL_REQEUST_TYPE,
        ML_CTRL_VALUE,
        ML_CTRL_INDEX,
        buf,
        sizeof(buf),
        HZ * 5);

if (retval  0)
{
    DBG_ERR("usb_control_msg failed (%d)", retval);
    goto unlock_exit;
}

The command cmd is inserted into the buffer buf containing the data to be sent to the device. If the URB completes successfully, the corresponding handler is executed. It performs nothing fancy, except telling the driver that we launched a (yet uncorrected) command via the write syscall:

static void ml_ctrl_callback(struct urb *urb, struct pt_regs *regs)
{
    struct usb_ml *dev = urb-context;
    dev-correction_required = 0;
}

We do not want the missile launcher hardware to be damaged by neither sending improper commands nor sending any commands when it reached an axis boundary. Ideally, whenever an axis boundary is reached (meaning that the missile launcher cannot turn further in one direction), the device should stop the movement in the particular direction. The completion handler of the interrupt URB turns out to be the right place to implement this idea:

static void ml_int_in_callback(struct urb *urb, struct pt_regs *regs)
{
        /* ... */

        if (dev-int_in_buffer[0]  ML_MAX_UP  dev-command  ML_UP)
        {
            dev-command = ~ML_UP;
            dev-correction_required = 1;
        } else if (dev-int_in_buffer[0]  ML_MAX_DOWN 
                dev-command  ML_DOWN)
        {
            dev-command = ~ML_DOWN;
            dev-correction_required = 1;
        }

        if (dev-int_in_buffer[1]  ML_MAX_LEFT  dev-command  ML_LEFT)
        {
            dev-command = ~ML_LEFT;
            dev-correction_required = 1;
        } else if (dev-int_in_buffer[1]  ML_MAX_RIGHT 
                dev-command  ML_RIGHT)
        {
            dev-command = ~ML_RIGHT;
            dev-correction_required = 1;
        }

        /* ... */
}

The above code is used to set the correction_required variable which triggers a “correction” control URB: this URB contains simply the last command without the harming bit. Remember that the URB callback functions run in interrupt context and thus should not perform any memory allocations, hold semaphores, or cause anything putting the process to sleep. With this automatic correction mechanism, the missile launcher is shielded from improper use. Again, it does not impose policy constraints, it protects only the device.

Controlling the missile launcher from user-space

For most folks fun starts in here. One doesn’t kick the bucket when dereferencing NULL-pointers and the good old libc is available, too. After having loaded the kernel module, the missile launcher is accessible via /dev/ml0. A second missile launcher would show up as /dev/ml1 and so on. Here is a very simple application to control the device:

#include fcntl.h
#include stdio.h
#include stdlib.h
#include unistd.h

#define DEFAULT_DEVICE      "/dev/ml0"
#define DEFAULT_DURATION    800

#define ML_STOP         0x00
#define ML_UP           0x01
#define ML_DOWN         0x02
#define ML_LEFT         0x04
#define ML_RIGHT        0x08
#define ML_FIRE         0x10

#define ML_FIRE_DELAY   5000

void send_cmd(int fd, int cmd)
{
    int retval = 0;

    retval = write(fd, cmd, 1);
    if (retval  0)
        fprintf(stderr, "an error occured: %d\n", retval);
}

static void usage(char *name)
{
    fprintf(stderr,
            "\nusage: %s [-mslrudfh] [-t msecs]\n\n"
            "  -m      missile launcher [/dev/ml0]\n"
            "  -s      stop\n"
            "  -l      turn left\n"
            "  -r      turn right\n"
            "  -u      turn up\n"
            "  -d      turn down\n"
            "  -f      fire\n"
            "  -t      specify duration in milli seconds\n"
            "  -h      display this help\n\n"
            "notes:\n"
            "* it is possible to combine the directions of the two axes, e.g.\n"
            "  '-lu' send_cmds the missile launcher up and left at the same time.\n"
            "" , name);
    exit(1);
}

int main(int argc, char *argv[])
{
    char c;
    int fd;
    int cmd = ML_STOP;
    int duration = DEFAULT_DURATION;
    char *dev = DEFAULT_DEVICE;

    if (argc  2)
        usage(argv[0]);

    while ((c = getopt(argc, argv, "mslrudfht:")) != -1)
    {
        switch (c)
        {
            case 'm': dev = optarg;
                      break;
            case 'l': cmd |= ML_LEFT;
                      break;
            case 'r': cmd |= ML_RIGHT;
                      break;
            case 'u': cmd |= ML_UP;
                      break;
            case 'd': cmd |= ML_DOWN;
                      break;
            case 'f': cmd = ML_FIRE;
                      break;
            case 's': cmd = ML_STOP;
                      break;
            case 't': duration = atoi(optarg);
                      break;
            default: usage(argv[0]);
        }
    }

    fd = open(dev, O_RDWR);
    if (fd == -1)
    {
        perror("open");
        exit(1);
    }

    send_cmd(fd, cmd);

    if (cmd  ML_FIRE)
        duration = ML_FIRE_DELAY;
    else if (cmd == ML_UP || cmd == ML_DOWN)
        duration /= 2;
    usleep(duration * 1000);

    send_cmd(fd, ML_STOP);

    close(fd);

    return EXIT_SUCCESS;
}

This tool, let’s name it ml_control, allows the user to send data to the device via the write syscall. For example, the device moves three seconds up and left with ./ml_control -ul -t 3000, shoots with ./ml_control -f, or stop with ./ml_control -s. Consider the code as proof of concept, of course more sophisticated applications are imaginable.

Summary

In this article, I frame the creation of a USB device driver for the Linux kernel. At first I reverse-engineer the unknown USB protocol by intercepting all USB traffic to and from the device with the Windows driver. Having captured the complete communication primitives, I explain how to build a USB kernel driver. Finally, a proof-of-conecpt user-space tool is presented that lays the foundation stone for further fancy ideas. Future work touches topics like augmenting the missile launcher with a video camera or mounting it on arbitrary devices. The code from this article and a full implementation of the device driver is available at my github repository.

Examining and Dissecting tcpdump/libpcap Traces

2007-01-06T08:00:00Z

Almost every network research involves trace-based analysis with a captured stream of packets. This article presents a set of analysis tools which extract detailed information from tcpdump/libpcap traces.

Having a pcap trace in place, it is about to gather futher details from it, e.g., the traffic peak rate or which protocol accounts for the largest share. Many tools exist to accomplish this task. Below I will describe three tools that particularly caught my attention while working with network traffic: tcpdstat, ipsumdump, and Netdude.

tcpdstat

Written by Kenjiro Cho, tcpdstat is a powerful tool that performs an in-depth protocol breakdown by bytes and packets. It further displays average and maximum transfer rates, IP flow information, and packet size distribution. Dave Dittrich applied several tweaks the tool to support a broader range of protocols and services, and to report more details about flow rates.

Here is an example output (of Dave’s enhanced version):

DumpFile:  trace.pcap
FileSize: 98876.89MB
Id: 200703011241
StartTime: (anonymized)
EndTime:   (anonymized)
TotalTime: 7216.13 seconds
TotalCapSize: 96826.91MB  CapLen: 1514 bytes
# of packets: 134347439 (96826.91MB)
AvgRate: 113.10Mbps  stddev:47.96M   PeakRate: 260.92Mbps

### IP flow (unique src/dst pair) Information ###
# of flows: 1612801  (avg. 83.30 pkts/flow)
Top 10 big flow size (bytes/total in %):
 33.6%  3.2%  2.2%  1.5%  1.4%  1.0%  1.0%  0.9%  0.8%  0.8%

### IP address Information ###
# of IPv4 addresses: 480065
Top 10 bandwidth usage (bytes/total in %):
 34.4% 34.4%  3.3%  3.3%  3.0%  2.7%  2.3%  1.8%  1.5%  1.5%

### Packet Size Distribution (including MAC headers) ###
<<<<
 [   32-   63]:   20839652
 [   64-  127]:   38798140
 [  128-  255]:    3947049
 [  256-  511]:    3746280
 [  512- 1023]:    5675556
 [ 1024- 2047]:   61340762
>>>>

### Protocol Breakdown ###
<<<<
     protocol           packets                 bytes           bytes/pkt
------------------------------------------------------------------------
[0] total        134347439 (100.00%)     101530372750 (100.00%)    755.73
[1] ip           134347439 (100.00%)     101530372750 (100.00%)    755.73
[2]  tcp         118172509 ( 87.96%)      97361936181 ( 95.89%)    823.90
[3]   ftpdata        18640 (  0.01%)         16529412 (  0.02%)    886.77
[3]   ftp            72372 (  0.05%)          4697330 (  0.00%)     64.91
[3]   ssh         13849679 ( 10.31%)      11113777353 ( 10.95%)    802.46
[3]   telnet          9007 (  0.01%)          1526445 (  0.00%)    169.47
[3]   smtp         2133471 (  1.59%)       1447293494 (  1.43%)    678.38
[3]   name              23 (  0.00%)             1426 (  0.00%)     62.00
[3]   dns            35071 (  0.03%)          7071657 (  0.01%)    201.64
[3]   http(s)     25043480 ( 18.64%)      30677552254 ( 30.22%)   1224.97
[3]   http(c)     16165378 ( 12.03%)       2182851897 (  2.15%)    135.03
[3]   kerb5            370 (  0.00%)            30610 (  0.00%)     82.73
[3]   pop3           82382 (  0.06%)         26718043 (  0.03%)    324.32
[3]   sunrpc            30 (  0.00%)             3002 (  0.00%)    100.07
[3]   ident           5107 (  0.00%)           322074 (  0.00%)     63.07
[3]   nntp            1262 (  0.00%)           292679 (  0.00%)    231.92
[3]   epmap         209144 (  0.16%)         12909976 (  0.01%)     61.73
[3]   netb-se       404237 (  0.30%)         47178014 (  0.05%)    116.71
[3]   imap          125983 (  0.09%)        100889454 (  0.10%)    800.82
[3]   bgp              482 (  0.00%)            43139 (  0.00%)     89.50
[3]   ldap            7131 (  0.01%)          1434769 (  0.00%)    201.20
[3]   https        2941177 (  2.19%)       1802114169 (  1.77%)    612.72
[3]   ms-ds         245214 (  0.18%)         24263111 (  0.02%)     98.95
[3]   rtsp         1023246 (  0.76%)        691696863 (  0.68%)    675.98
[3]   ldaps           2828 (  0.00%)           209272 (  0.00%)     74.00
[3]   socks           7883 (  0.01%)          1340672 (  0.00%)    170.07
[3]   kasaa          13348 (  0.01%)          1124944 (  0.00%)     84.28
[3]   mssql-s       309786 (  0.23%)         20411848 (  0.02%)     65.89
[3]   squid          51381 (  0.04%)         14079861 (  0.01%)    274.03
[3]   ms-gc           1865 (  0.00%)           493682 (  0.00%)    264.71
[3]   ms-gcs          2034 (  0.00%)           481178 (  0.00%)    236.57
[3]   hotline            6 (  0.00%)              682 (  0.00%)    113.67
[3]   realaud        19784 (  0.01%)         13197979 (  0.01%)    667.10
[3]   icecast       390203 (  0.29%)        291651836 (  0.29%)    747.44
[3]   gnu6346         6324 (  0.00%)          1048473 (  0.00%)    165.79
[3]   gnu6348          342 (  0.00%)            26047 (  0.00%)     76.16
[3]   gnu6349           14 (  0.00%)             2767 (  0.00%)    197.64
[3]   gnu6350            4 (  0.00%)              732 (  0.00%)    183.00
[3]   irc6666            7 (  0.00%)              434 (  0.00%)     62.00
[3]   irc6667         1379 (  0.00%)           196155 (  0.00%)    142.24
[3]   irc6668            2 (  0.00%)              124 (  0.00%)     62.00
[3]   irc6669            9 (  0.00%)              666 (  0.00%)     74.00
[3]   napster           21 (  0.00%)             1344 (  0.00%)     64.00
[3]   irc7000            7 (  0.00%)              824 (  0.00%)    117.71
[3]   http-a        129807 (  0.10%)         71136838 (  0.07%)    548.02
[3]   other       54862568 ( 40.84%)      48787331392 ( 48.05%)    889.26
[2]  udp          13069221 (  9.73%)       3895596348 (  3.84%)    298.07
[3]   name              18 (  0.00%)             1989 (  0.00%)    110.50
[3]   dns          1799081 (  1.34%)        264263480 (  0.26%)    146.89
[3]   kerb5            100 (  0.00%)            25812 (  0.00%)    258.12
[3]   sunrpc           581 (  0.00%)            57157 (  0.00%)     98.38
[3]   ntp            50387 (  0.04%)          4534933 (  0.00%)     90.00
[3]   epmap             17 (  0.00%)             1824 (  0.00%)    107.29
[3]   netb-ns       148619 (  0.11%)         14736588 (  0.01%)     99.16
[3]   netb-se         1272 (  0.00%)           328673 (  0.00%)    258.39
[3]   ms-ds              8 (  0.00%)              883 (  0.00%)    110.38
[3]   kazaa             29 (  0.00%)             3546 (  0.00%)    122.28
[3]   mssql-s           44 (  0.00%)             3832 (  0.00%)     87.09
[3]   mcast        7216682 (  5.37%)       1943012688 (  1.91%)    269.24
[3]   realaud       459195 (  0.34%)        273532235 (  0.27%)    595.68
[3]   halflif           81 (  0.00%)             5890 (  0.00%)     72.72
[3]   starcra           45 (  0.00%)             6367 (  0.00%)    141.49
[3]   everque            9 (  0.00%)             1351 (  0.00%)    150.11
[3]   unreal          1066 (  0.00%)            93951 (  0.00%)     88.13
[3]   quake             20 (  0.00%)             1860 (  0.00%)     93.00
[3]   other        3384119 (  2.52%)       1394472416 (  1.37%)    412.06
[2]  icmp          3105709 (  2.31%)        272840221 (  0.27%)     87.85
[2]  frag            30903 (  0.02%)         25672129 (  0.03%)    830.73
>>>>

At first, tcpdstat shows a summary including the trace size, time, and number of packets. Most interesting here is the average and peak traffic rate in Mbps and the standard deviation, measuring how bursty the traffic was. The second block shows the number of IP flows (i.e., the pair of src and dst address) and the 10 biggest flows in percent. Next, the total disctinct IP addresses are displayed along with the top 10 bandwith usage in percent. Unfortunately, only the percent values are shown and not the addresses themselves. The next block displays the packet size distribution including MAC headers. The last big block is the very informative, illustrating a per-packet and per-byte protocol breakdown. The IP protocol is partitioned in TCP, UDP, ICMP, and fragmented packets. TCP and UDP packets are further broken down into their application layer protocols.

Tcpdstat is great to get a bunch of information out of a trace, very easy to use, but it lacks some flexibility. The tool is your perfect choice if the above output is enough for you. Otherwise, you might incorporate further tools into your analysis.

ipsumdump

The ipsumpdump utility from Eddie Kohler is the “swiss-army knife” for trace analysis. Itsummarizes TCP/IP dump files into a nice ASCII format that can be intuitively understood and easily parsed.

Ipsumdump can read packets from network interfaces, from tcpdump files, and from existing ipsumdump files. It will transparently uncompress tcpdump or ipsumdump files when necessary. It can randomly sample traffic, filter traffic based on its contents, anonymize IP addresses, and sort packets from multiple dumps by timestamp. Also, it can optionally create a tcpdump file containing actual packet data.

To demonstrate its versatility, I provided the output of ipsumdump-h below:

'Ipsumdump' reads IP packets from tcpdump(1) files, or network interfaces,
and summarizes their contents in an ASCII log.

Usage: ipsumdump [CONTENT OPTIONS] [-i DEVNAMES | FILES] > LOGFILE

Options that determine summary dump contents (can give multiple options):
  -t, --timestamp            Include packet timestamps.
  -T, --first-timestamp      Include flow-begin timestamps.
  -s, --src                  Include IP source addresses.
  -d, --dst                  Include IP destination addresses.
  -S, --sport                Include TCP/UDP source ports.
  -D, --dport                Include TCP/UDP destination ports.
  -l, --length               Include IP lengths.
  -p, --protocol             Include IP protocols.
      --id                   Include IP IDs.
  -g, --fragment             Include IP fragment flags ('F' or '.').
  -G, --fragment-offset      Include IP fragment offsets.
      --ip-sum               Include IP checksums.
      --ip-opt               Include IP options.
  -F, --tcp-flags            Include TCP flags word.
  -Q, --tcp-seq              Include TCP sequence numbers.
  -K, --tcp-ack              Include TCP acknowledgement numbers.
  -W, --tcp-window           Include TCP receive window (unscaled).
  -O, --tcp-opt              Include TCP options.
      --tcp-sack             Include TCP selective acknowledgement options.
      --udp-length           Include UDP lengths.
  -L, --payload-length       Include payload lengths (no IP/UDP/TCP headers).
      --payload              Include packet payloads as quoted strings.
      --payload-md5          Include MD5 checksum of packet payloads.
      --capture-length       Include lengths of captured IP data.
  -c, --packet-count         Include packet counts (usually 1).
      --link                 Include link numbers (NLANR/NetFlow).

Data source options (give exactly one):
  -r, --tcpdump              Read tcpdump(1) FILES (default).
  -i, --interface            Read network devices DEVNAMES until interrupted.
      --ipsumdump            Read existing ipsumdump FILES.
      --format FORMAT        Read ipsumdump FILES with format FORMAT.
      --dag[=ENCAP]          Read DAG-format FILES.
      --nlanr                Read NLANR-format FILES (fr/fr+/tsh).
      --netflow-summary      Read summarized NetFlow FILES.
      --tcpdump-text         Read tcpdump(1) text output FILES.

Other options:
  -o, --output FILE          Write summary dump to FILE (default stdout).
  -b, --binary               Create binary output file.
  -w, --write-tcpdump FILE   Also dump packets to FILE in tcpdump(1) format.
  -f, --filter FILTER        Apply tcpdump(1) filter FILTER to data.
  -A, --anonymize            Anonymize IP addresses (preserves prefix & class).
      --no-promiscuous       Do not put interfaces into promiscuous mode.
      --bad-packets          Print '!bad' messages for bad headers.
      --sample PROB          Sample packets with PROB probability.
      --multipacket          Produce multiple entries for a flow identifier
                             representing multiple packets (NetFlow only).
      --collate              Collate packets from data sources by timestamp.
      --interval TIME        Stop after TIME has elapsed in trace time.
      --limit-packets N      Stop after processing N packets.
      --map-address ADDRS    When done, print to stderr the anonymized IP
                             addresses and/or prefixes corresponding to ADDRS.
      --record-counts TIME   Record packet counts every TIME seconds in output.
      --random-seed SEED     Set random seed to SEED (default is random).
      --no-mmap              Don't memory-map input files.
  -q, --quiet                Do not print progress bar.
      --config               Output Click configuration and exit.
  -V, --verbose              Report errors verbosely.
  -h, --help                 Print this message and exit.
  -v, --version              Print version number and exit.

Report bugs to <kohler@cs.ucla.edu>.

Among the above options, some deserve more attention. For example, the --payload-md5 option includes a MD5 checksum (e.g. i4CxGSojVHB2XcZw97ZpQb) of the packet payload in the dump. This option comes in handy when you want to check for packet duplicates. Another nifty option is --anonymize. Since traces can contain sensitive data, it is possible to anonymize the IP addresses in the output in order to prevent information leakage. The anonymization preserves prefix and class. In high-volume networks, the --sample=p option might be interesting. It samples packets with probability p. That is, p is the chance that a packet will cause output to be generated. The actual probability may differ from the specified probability, due to fixed point arithmetic. If you want to merge several trace files retaining the temporal order, --collate together with --write-tcpdump sorts your packets with increasing timestamp.

Summing up, ipsumdump is a flexible trace analysis tool complementing tcpdump. However, it does not feature predefined evaluation methods like [tcpdstat][tcpdstat] (which is actually not a design goal). Nevertheless, ipsumdump is a valuable tool to quickly generate easily readable ASCII output or to obtain well-formatted output for subsequent processing/scripting. It unveils its real power when used for trace manipulation such as merging, modifying, or anonymizing tcpdump traces.

Netdude

Netdude (Network dump data displayer and editor) is a graphical tool to edit tcpdump trace files, written by Christian Kreibich. In fact, it is a front-end to the libnetdude packet manipulation library. Since complex trace manipulation is non-trivial and often requires custom coding, Netdude provides a GUI enabling users to

edit traces of arbitrary size in a scalable fashion. Netdude never loads more than a configurable maximum number of packets into memory at any time.
edit multiple traces at the same time, making it easy to move packets from one trace to a different one.
Modify every field in protocol headers for which a protocol plugin provides support. These modifications can be applied to either only individually selected packets, packets currently in memory, or all packets in the trace, including the ones not currently loaded.
Filter packets by using filter plugins. Netdude 0.4.6 ships with a BPF filter plugin that allows you to use the standard BPF filter language to define your filters.
Inspect and edit raw packet content using Netdude’s payload editor in either hex or ASCII mode whichever is more convenient for the payload you are editing.
Move packets around, duplicate them, remove them from traces.
See the tcpdump output updating instantly according to the modifications you’re making.
Conveniently use the clipboard to select lines from the tcpdump output for situations when you need the tcpdump output only (e.g., when writing documentation, papers or emails).

As soon as packet editing is part of the game, Netdude offers an interactive and user-friendly GUI to perform even sophisticated manipulations. The screenshot below illustrates how one can graphically manipulate the TCP header fields.

Conclusion

Nowadays, captured network traffic is mostly available as a tcpdump/libpcap trace. In this article, I introduce three tools enabling in-depth trace examination. At first, the tool tcpdstat provides a high-level view of the trace ingredients with a detailed protocol breakdown. Second, ipsumdump offers a flexible means to generate nicely formatted ASCII dump. It can be used to quickly extract a desired piece of information or as a multi-purpose output generator. Finally, Netdude features a comfortable GUI to selectively manipulate packet details. Equipped with this arsenal, trace-based network analysis feels like a hot knife through butter.

Matthias Vallentin

High-Level Network Traffic Profiling

Better NIDS Performance by Tracking Roaming Users

Passive Detection

Reducing False Positives

Summary

Omni, Take Two

A Garden Variety of Bloom Filters

Introduction

Bloom Filters

Terminology

Basic

Multisets

Counting

Bitwise

Spectral

Aging

Stable

A2

libBf

Related Work

Analyzing Facebook Webchat Sessions with Bro

The Facebook Webchat Protocol

The Bro Script

Taming the Sheep: Detecting Sidejacking with Bro

Detection

Design Considerations

Decision Trees

Operation

Mitigation

Bro Code

Probability and Statistics Cheat Sheet

Screenshots

Should You Trust Your SSL Certificate?

Proactive Protection

Trust-On-First-Use

Ground Truth

False Negatives

User Failure

Retrospective Analysis

Email Attachment Processing with Bro

The Doom of Client-Side Wireless Network Security

Writing a Linux Kernel Driver for an Unknown USB Device

Introduction

Related Work

USB primer

Reverse-engenireering the USB protocol

The device driver

Controlling the missile launcher from user-space

Summary

Examining and Dissecting tcpdump/libpcap Traces

tcpdstat

ipsumdump

Netdude

Conclusion

A²