Inadvertent Menace to Technology

2011-10-15T12:20:00.000-07:00

Verified by VISA, unverified by the user...

Purchasing something on New Egg...after punching in my new credit card info, I get a redirect claiming to be "Verified by Visa", telling me to input my birth date, last 4 of my social security, etc. This is suspicious. Then there's the url:

https://www.verifiedbyvisa.com/VisaGold/app/pahandler.lt?vga= %3D%3Ahttps%3A%2F%2Fsecure2.arcot.com%2Facspage%2Fcap.cgi

Even if verified by visa and arcot are legit sites, they are using bogus means. I'm not sure what I'm going to do here. Could they make this mechanism look any less legit? Maybe some banner ads for pharmacee?

2011-05-10T00:40:00.000-07:00

Trouble with VNC - shift key not transmitted

I ran into this recently: after launching a VNC server on my work machine, I couldn't log in because it wasn't accepting the shift key. Essentially, no capital letters and half the punctuation was unavailable. This made it so I couldn't log on because my password is full of mixed case and punctuation.

Thanks to bramschoenmakers.nl for the fix. I'm only putting this here so I can find it again if needed!

http://www.bramschoenmakers.nl/en/node/714

invoke x11vnc with -xkb flag or add the line xkb to your ~/.x11vncrc

2010-10-02T23:46:00.000-07:00

Ubuntu gdm problem solved.

I ran into this shortly after updating 9.10, got the black box in place of login screen, following log messages in /var/log/gdm/:0-greeter.log

(gnome-settings-daemon:3419): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed

(gnome-settings-daemon:3419): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
Window manager warning: Failed to read saved session file /var/lib/gdm/.config/metacity/sessions/10a9a99fd3e4ce7c9b128608721774271800000034130008.ms: Failed to open file '/var/lib/gdm/.config/metacity/sessions/10a9a99fd3e4ce7c9b128608721774271800000034130008.ms': No such file or directory

/etc/motd told me system load was over 5, saw 4-5 gdm processes each eating 15-27% cpu.

I figured it was a faulty update, possibly of my evil binary nvidia driver. Turns out /tmp was full! Clearing it solved the problem.

2009-06-25T11:24:00.000-07:00

Me: I hate the following as verbs: "partner", "dialogue". On the fence about "impact"

Jason: I can't stand it when people use "disconnect" as a noun. It's a freaking verb.

Me: There's a disconnect with you and that usage. Maybe we should partner to dialogue about the issue going forward.

Jason: Positively heinous. Nicely done.

2007-12-20T23:19:00.001-08:00

Generic Snort Advice:

I got an email about the Snort User Group that I had run in the past. Local interest waned, and I got a new job. Once in a while I get a contact via email. Here's my recent response:

I don't know your setup or the threats you face, but my generic advice is to place Snort on a separate box inside the firewall, so it doesn't have to analyze traffic the firewall will block. The general consensus is that there's too much scanning, worm attacks, etc. for the data outside the firewall to be any use. There won't be much stuff where you'll say, "Aha! I gotta do something about that!" There will be enough stuff inside the firewall to worry about. The firewall logs will give you that data if you have the cycles to do something with it.

One reason to use a separate box: a problem with Snort won't sever your internet connection. There are always problems with host management, like running out of disk space or hardware failure, and Snort has had a few problems itself.

Other generic advice: I found the major problem with doing Intrusion Detection was efficiently processing alerts. For my money, nothing beats the sguil console. It's awesome, and it's free. I would not bother with Arcsight, or Cisco's SIM console even if they were free. Sguil puts all the things you need to see over and over right on one page. And when you need to go deeper, you have the whole tcpdump. When you need to see what else might have happened, you have all the session data. Easy asci protocol decodes so you can see the http sessions. There's an IRC channel for it on Freenode, #snort-gui. The primary developer is on there most days, and is incredibly tolerant of n00bs. Martin Roesch and Richard Bejtlich are among the other notables who can be found there. There are VMware appliances available to try it out. Setup is a little daunting, but if you stick it out you'll be really glad you did. http://sguil.sourceforge.net

Another issue is tuning Snort, which is an ongoing maintenance issue. Shutting off noisy sources of false-positives is a big job, and you don't want to start over every time you update the ruleset. There's a script called Oinkmaster which makes it a lot easier. Make your changes to the oinkmaster file, and apply it to each new ruleset. Suppression directives are another supplemental approach; they let you ignore results for particular hosts for particular rules, while leaving the rule in force for everything else.

There's a lot of information out there. Let me know if you have questions and I'll try to point you to some of it.

2007-04-30T15:14:00.000-07:00

Living in Retardoville

We are going to set up an MS-SQL server at work. It's going to run on a dedicated machine, which gives better performance. We've identified a physical box to run this on, once we migrate what's currently running on that box to a virtual machine. This box is a dual processor server. MS licenses MS-SQL per processor.

So we are going to pull a processor from the machine to save licensing costs.

Now, what makes this retarded is not what we're doing, it's the stupidity and overhead of the MS revenue model.

We're not using a dedicated server strictly for performance and then electing to lobotomize it because we're morons. We're lobotomizing it to save money because the MS way gives us a financial incentive to cripple our server.

Go, Open Source!

2007-04-13T13:28:00.000-07:00

Installing Veritas Storage Foundation on CentOS 4.4

SF Basic is free, with a limit of 4 volumes. Offers multipathing, storage virtualization, snapshots, remote duplication, etc. Might be easier to get these advanced features with this than with an Open Source project like iSCSI Enterprise Target.

CAVEATS: Saw a blog entry complaining about terrible performance on linux.

Only installs on Red Hat Enterprise Linux, and Novell's brand. I searched for a record of anyone installing it on the RHEL free clone CentOS and came up basically empty. So FWIW, here is how I got it to install. I make no claims for the correctness of my perl; there may be better ways to accomplish what I did.

Experimental Installation on CentOS release 4.4 (Final) on this platform:

2.6.9-42.ELsmp #1 SMP Tue Aug 15 10:35:26 BST 2006 x86_64 x86_64 x86_64 GNU/Linux

tar -xzvf Q18982H.sfbasic.5.0.00.0.rhel4_x86_64.tar.gz
cd rhel4_x86_64/storage_foundation_basic/

There are problems with the install script installsf - edit a copy to match the following, or just paste this into your own myinstallsf (you may have to tweak the formatting):

#!/bin/sh

MYDIR1="/root/rhel4_x86_64/storage_foundation_basic/scripts/"

MYDIR="/root/rhel4_x86_64/perl/Linux/perl/lib/site_perl/5.8.8/"

MYOTHERDIR="/root/rhel4_x86_64/perl/Linux/perl/lib/site_perl/5.8.8/i686-linux-thread-multi-64int/"

DIRNAME=dirname $0

[ -z "DIRNAME" ] && DIRNAME="."

PDIR=/usr IOPT="/opt/VRTSperl/lib/site_perl/UXRT5.0"

if [ -z "$PDIR" ]; then

echo "Cannot find perl to execute $0" exit 1

fi

PERLVERS=$PDIR/bin/perl -v | grep This.is | awk '{print $4}' | sed 's/v//'

PERLFOR=$PDIR/bin/perl -v | grep This.is | awk '{print $7}'

IOPT="-I$IOPT -I$PDIR/lib/$PERLVERS/$PERLFOR -I$PDIR/lib/$PERLVERS -I$PDIR/lib/site_perl/$PERLVERS/$PERLFOR -I$PDIR/lib/site_perl/$PERLVERS -I$PDIR/lib/site_perl -I$MYDIR1 -I$MYDIR -I$MYOTHERDIR"

eval 'exec $PDIR/bin/perl -x -S $IOPT $0 ${1+"$@"}'

if 0;

#!perl

use strict "vars";

use warnings;

no warnings qw(uninitialized redefine internal);

use CPI::common::CPI;

use CPI::train::common::UXRT;

use CPI::train::Linux::UXRT;

CPI::main();

There's a library missing. Fix via:

yum install libaio

The installation script you prepared will now run, but fail. The error message is:

Undefined subroutine &CPI::pkg::Linux::VRTSpbx::pl_die called at /root/rhel4_x86_64/storage_foundation_basic/scripts//CPI/pkg/Linux/VRTSpbx.pm line 25,

To resolve this error, edit scripts/CPI/pkg/Linux/VRTSpbx.pm

change line 25: change pl_die --> CPI::pl_die

The installation script you prepared will run further, but still fail, with a reference to a missing rpm and a suggestion to install 'compat'. This is fixed by:

yum install compat-libstdc++-296.i386

Installation proceeds successfully, I think. In one of life's ironies, it took too long to work this out; I don't have time to experiment with it. Maybe later...

Here's the installation record/output.

------------------------------------

* Storage Foundation Basic 5.0 Installation Program

Copyright (c) 2006 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

The Licensed Software and Documentation are deemed to be "commercial computer software" and "commercial computer software documentation" as defined in FAR Sections 12.212 and DFARS Section 227.7202.

Logs for myinstallsf are being created in /var/tmp/myinstallsf-aBgSrT.

Enter the system names separated by spaces on which to install SF Basic: santest

Initial system check:

* Checking SF Basic installation on santest ............... not installed
* Checking libaio rpm ......................................... installed
* Checking glibc rpm .......................................... installed
* Checking kernel release on santest ................................. 42
* Checking distribution match with santest ........................... OK
* Checking architecture on santest ................................... OK
* Checking rpm dist match with santest ............................... OK
* Checking for SE Linux on santest ................................... OK
o Storage Foundation Basic 5.0 Installation Program

Checking system licensing

XXXXXXXXXXXXXXXXXXX successfully registered on santest
<<>>

Checking installed rpms on santest

SF Basic can be installed without optional rpms to conserve disk space.

* 1) Required Storage Foundation Basic rpms - 324 MB required 2) All Storage Foundation Basic rpms - 438 MB required

Select the rpms to be installed on all systems? [1-2,q,?] (2) 2

* The following SF Basic rpms will be installed:
* VRTSperl
* Veritas Perl 5.8.8 Redistribution
* VRTSvlic Veritas Licensing
* VRTSicsco Symantec Infrastructure Core Services Common
* VRTSpbx Symantec Private Branch Exchange
* VRTSsmf Symantec Service Management Framework
* VRTSatClient Symantec Product Authentication Service Client
* VRTSatServer Symantec Product Authentication Service
* VRTSobc33 Veritas Enterprise Administrator Core Service
* VRTSob Veritas Enterprise Administrator Service
* VRTSobgui Veritas Enterprise Administrator
* VRTSccg Veritas Enterprise Administrator Central Control Grid
* VRTSmh Veritas Storage Foundation Managed Host by Symantec
* VRTSaa Veritas Enterprise Administrator Action Agent
* VRTSspt Veritas Software Support Tools
* SYMClma Symantec License Inventory Agent
* VRTSvxvmcommon Veritas Volume Manager Common Package
* VRTSvxvmplatform Veritas Volume Manager Platform Specific Package
* VRTSdsa Veritas Datacenter Storage Agent
* VRTSfspro Veritas File System Management Services Provider Press
* [Enter] to continue:
* ..continued:
* VRTSvmdoc Veritas Volume Manager Documentation
* VRTSvmman Veritas Volume Manager Manual Pages
* VRTSlvmconv Veritas Linux LVM to VxVM Converter
* VRTSvdid Veritas Device Identification API
* VRTSddlpr Veritas Device Discovery Layer Services Provider
* VRTSvmpro Veritas Volume Manager Management Services Provider
* VRTSvsvc Veritas Volume Server and Client Provider
* VRTSdcli Veritas Distributed Command Line Interface
* VRTSalloc Veritas Volume Manager Intelligent Storage Provisioning
* VRTSvxfscommon Veritas File System Common package
* VRTSvxfsplatform Veritas File System Platform Specific Package
* VRTSfsman Veritas File System Manual Pages
* VRTSfsdoc Veritas File System Documentation
* VRTSfssdk Veritas File System Software Developer Kit
* VRTSfsmnd Veritas File System Software Developer Kit Manual Pages
* VRTSvxmsa Veritas Mapping Service, Application Libraries
* VRTSmaprocommon Veritas Storage Foundation GUI for Mapping Press
* [Enter] to continue:
o Checking for patch(1) rpm on santest .......... version 2.5.4 installed
o It is possible to install SF Basic rpms without performing configuration. It is optional to configure SF Basic now. If you choose to configure SF Basic later, you can either do so manually or run the installsf -configure command. Are you ready to configure SF Basic? [y,n,q] (y) y
o Installing SF Basic: 100%
The following rpms failed to install on santest:
o VRTSddlpr
o VRTSvsvc
o VRTSdcli
o VRTSalloc
o VRTSmapro-common
o The enclosure-based naming scheme is a feature of Veritas Volume Manager. It allows one to reference disks using a symbolic name that is more meaningful than the operating system's normal device access name. This symbolic name is typically derived from the array name.
* . Do you want to set up the enclosure-based naming scheme? [y,n,q,?] (n) y
* . Veritas Volume Manager default disk group name configuration: Many Veritas Volume Manager commands affect the contents or configuration of a disk group. Such commands require that the user specify a disk group. This is accomplished by using the -g option of a command or setting the VXVM_DEFAULTDG environment variable. An alternative to these two methods for some commands is to configure the name of the default disk group of a system. Note: The default disk group feature is not available with the DCLI vxadm(1M) and vxquery(1M) commands. When using these commands, a required disk group must be explicitly identified using the -g option.
* . Do you want to set up a default disk group for each system? [y,n,q,?] (y)
* . Specify a default disk group name for system santest. [?] admin
* . You have decided to specify the default disk group as follows:
o Host: santest ....................................... Disk group: admin
o Note: If 'nodg' is displayed, then the host will be configured to have no default disk group. Is this correct? [y,n,q] (y)
* . Verify the install systems Fully Qualified Hostnames.
o Querying fully qualified domain name of host "santest" ........... fail
Press [Enter] to continue: Unable to find FQHN for santest.sscc.storage! Enter the fully qualified hostname of "santest" or 'QUIT' to quit install.
* . Enter the fully qualified host name: santest.sscc.storage
* . Choose how this install will be managed. This product can configured one of the two following ways:
o Storage Foundation Management Server managed host (Recommended)
o Standalone host Pre-requisites for Storage Foundation Management Server managed host:
+ A Storage Foundation Management Server (SFMS) is configured
+ SFMS should be currently running and fully operational.
+ Authentication Broker that is used by SFMS should be running.
The following information would be requested during configuration:
+ The host name of the SFMS
+ Previously set SFMS agent account password
There are no pre-requisites for deploying as a standalone host.
Enabling Storage Foundation Management Server management simplifies and improves management of the complex data center resources, reducing planned and unplanned down time.
* . Enable Storage Foundation Management Server Management? [y,n,q] (y) n

Verify the install systems Fully Qualified Hostnames.

* Configuring VEA in STANDALONE mode on "santest" .................... ok
* Configuring gridnode on "santest" .................................. ok
* Registering gridnode on "santest" .................................. ok
* Configuring actionagent on "santest" ............................... ok
* Registering actionagent on "santest" ............................... ok

Registering StorageAgent on "santest" .............................. ok
* Do you want to start Storage Foundation Basic processes now? [y,n,q] (y)
* .
o Starting SF Basic: 100%
o Startup completed successfully on all systems
o Setting default disk group to admin on santest ................... Done
Installation log files, summary file, and response file are saved at:
o /opt/VRTS/install/logs/myinstallsf-aBgSrT
o Scalars leaked: -1
* Scalars leaked: 1
* . [root@santest storage_foundation_basic]#

2007-03-29T13:09:00.000-07:00

Preserve us from Sharepoint

It's a web server.

In a world blessed with Zope, Plone, wikis, and other content management systems, I'm amazed that anyone would pay money for Sharepoint. They should pay you to adopt it because of the hideous negative value that is vendor lock-in.

It's a web server.

It will cost a fortune to customize to your needs. Just as any other system will. Only you will continue paying for it until you die. Then someone else will pay for it. Unless they decide you are moving to whatever else they want to sell you. Then you will pay for that whatever else until you die, at which point someone else will take over paying for it.

It's a web server.

Ooh, it integrates with our hopelessly vendor-locked-in authentication system? (AD) pam_ldap, folks.

It's a web server.

It's a web server that will be bought by people who look around for Microsoft products to buy, rather than needs to address. Because if you'll buy that, you'll buy dog turds in a can so long as it comes from them.

2007-03-23T13:30:00.000-07:00

Repeat Rant: Do not Perform Arithmetic on Ordinal Numbers

I've previously written on the topic here .

Today I attended a meeting of the Agora, a monthly symposium on networks and security at the University of Washington. It was my first meeting and I was impressed. Lots of local infosec celebrities. The last speaker jumped on my last nerve (as my housemate puts it), though.

Andrew Macpherson presented a draft of an unpublished paper on a Threat Calculator he and others developed at the U of New Hampshire as part of an outfit called Justiceworks. The dude has some background; during 9/11 he was at Dartmouth College's Institute for Security Technology Studies and did some work for the Feds during the aftermath. Also in his favor is the correct use of the term "Threat" - at least per Richard Bejtlich, who is smarter than me and probably you. A threat is an entity with the intent and capability to attack.

Where he goes wrong is basically...his whole project. He takes a bunch of factors (24, I believe) to describe threat-actors (ok, he waters down the term "threat" - deduct a couple points), assigns each factor a rating on a scale of 1-5, and adds them up. Highest possible score is 140 points.

Where do I begin.

Numbers used to rank items are called Ordinal Numbers. Apart from Set Theory, you can't do arithmetic on ordinal numbers. Why not? Think about it. There is no precise relationship between numbers on a ranking scale. 1 < 2, but 1 is not half of 2 nor is it one less than 2. One is just in a position to the left on the scale. It might help to think of "First, Second, etc." instead of "One, Two, etc." We are so used to "regular" numbers that it's easy to make this mistake, but basing any decisions on this fundamentally flawed reasoning is a blunder. And the seeming precision - "hey, we quantified this shit! gives resonance to the blunder. You just painted a qualitative assessment with quantitative coloring.

You can't do arithmetic on numbers where the units are different. 5 gallons plus 3 hectares = ???

There's a problem of weight. Is a 5 for technical capability = a 5 for National/Cultural Stability? They have the same weight in the calculator. Funny that all 24 factors are exactly equal in importance.

There's a problem of scale. Even given the appropriateness of addition for this exercise, is a 5 really only 4 places more than a 1? This scale encompasses the PRC and my technologically illiterate neighbor. Sure, something close to 140 is scarier than something close to 24, but what's the scare curve look like? Does the pucker factor zoom skyward as you cross 100? 50?

These combine in unwholesome ways to undermine the whole exercise. The Netherlands scores high in technical aptitude, national stability, infrastructure similarity (they use similar control systems so they know how to attack them), etc. All of which is totally blown away by low scores in ideological antagonism. Yet overall they probably score higher than outfits that WILL commit serious cyber attacks some day.

Finally, there's the fundamental problem of pulling numbers out of your ass 24 times and then munging each together and gazing lovingly on the results like it means more than a single number pulled out of your ass. Some of those numbers are intensely debatable, such as those relating to the motivation of organized crime to conduct strategic cyber attacks. Maybe they will rise to that level of extortion some day. Maybe not. Who knows? I don't - and they don't either. Write me an assessment arguing for one or the other, and include your reasoning. Don't sink your reasoning in a morass of bogus numbers (and take any prospect for debate with it because you can't argue with something as scientifically sound as a number!).

So why does this matter? The tools we use to help us think affect the decisions we make. The pervasive use of Powerpoint at NASA helped doom the shuttle Columbia. (Search "Columbia Powerpoint" for a host of articles detailing the official findings of the accident report, which faulted NASA for discussing the risks via Powerpoint presentations rather than technical reports.) Bad tools, bad decisions. If common sense will rescue us from dumb conclusions reached by this approach, the exercise is merely meaningless. We should have just gone with common sense at the beginning. But I would be very worried if this process really drives any actual resource allocation. "Gambia got a higher rating than Senegal, so prep some wargames..."

2007-03-21T22:13:00.000-07:00

Sguil tricks: mass catagorization of events through database queries

If you don't want to catagorize events one by one (or aggregate by aggregate) via the sguil client, you can manipulate the database directly. This is useful if you have tens or hundreds of thousands of useless alerts. My personal best is 1.6 million.

The first step is to stop sguild. It prefers to think it's the only thing acting on the database at a given time, and will be confused if you alter the db while sguild is running.

Then you run a sql query as in the following example:

update event
set status = (desired code) where
status = (current code - usually 0 for this kind of task)
and (condition = whatever, eg signature = 'ICMP PING NMAP')
limit (some limit on the number of times you want this to run)
;

The status codes are a little tricky - I read the source of the sguil client to dig up the following:

"Cat I: Unauthorized Root Access" 11

"Cat II: Unauthorized User Access" 12

"Cat III: Attempted Unauthorized Access" 13

"Cat IV: Successful Denial of Service Attack" 14

"Cat V: Poor Security Practice or Policy Violation" 15

"Cat VI: Reconnaissance/Probes/Scans" 16

"Cat VII: Virus Infection" 17

So if you want to catagorize a real time event (status currently 0) as Cat VII, the query fragment is

update event set status = 17 where status = 0 and ....

I use a different system. I'm not interested in taxonomy, I'm interested in tasks. That is, do I have to deal with this or not? I am the remediator, if remediation is to be done. So I don't need to capture the type of incident. I KNOW that. I also won't run down every event, but I don't want to lie when I dismiss something without conclusive investigation. So I have two catagories for honest punts. This way they won't get buried in the False Positive sections.

My system is Cat I False Positive - no action required SQL update event set status = 11

Cat 2 False Positive - action required (tune rule, suppress alert, mitigate condition (i.e. reconfigure noisy host))set status = 12

Cat 3 True Positive - no action required (harmless worm attacking patched host)set status = 13

Cat 4 True Positive - action required (possibly should escalate F9)set status = 14

Cat 5 Not sure, not worried (Punt)set status = 15

Cat 6 Not sure, worried (Punt)set status = 16

Do not set status = 2 - that's escalate! This gets inserted into sguil client, slowing things down!

2007-03-13T12:39:00.000-07:00

Stoopid MS tricks #2

Applications should get their time from the operating system.

There, that wasn't so hard, was it?

Enter the change to Daylight Savings.

If you use Outlook, and set an appointment for 11:00, Outlook will change that. I can only think this is an implementation of "Do What I Mean" technology, which always fails.

11:00 is 11:00. There are no large or small values of 11:00. If I am to meet my boss at 11:00, that is the time I am to meet him. My appointment book should not adjust this. If my system clock is wrong, my reminder popups will pop up at the wrong time; that is my lookout.

When Congress changed the DST date, it caught a lot of software companies. Well, mostly Microsoft. Linux and the BSDs have a simple (extensive, but simple) Time Zone file. Tell the machine where you are, and it will adjust the local time based on that file. Simple. One slashdotter tried to make the case that Windows was easier to use because all he had to do was apply a patch to the OS, a patch to the application, and a patch to the Exchange server and he was done. He lied, as I'll describe below. But he tried to contrast updating the Time Zone file on his linux server. He had to do it in three places, because he was running two servers with chroot jails. That's a feature where you can isolate a service to it's own file system. If the service is hacked, the attacker is potentially limited to that file system and can't make mischief elsewhere. I'm not sure why you'd complain about having the option to use a security feature that ISN'T EVEN AVAILABLE on Windows.

Back to Windows.

There is no DST patch for Windows 2000 - you have to manually edit the registry. There are Windows fanboys who say it's impossible to maintain a system if you have to manually edit a text based config file. Manually editing a convoluted, monolithic database designed by idiots is fine, though.

If you do apply the patch to OS, application, and server, your appointments between 3/11/2007 and 4/1/2007 will get adjusted and will be in an unpredictable state. If you are using Windows XP and Office XP, and do not apply the patches, you will be in a predictably bad state as the appointments get moved an hour ahead. Why? Did 11:00 suddenly become something different? Did we want to meet at UTC -7 rather than 11:00 ? I ENTERED 11:00 AND THAT'S WHAT I MEANT.

Even if it gets things right, it will resend meeting invitations for some goddam reason.

Deeply disturbing and disturbed.

Let's say it again: applications, if you need to know what time it is, ask the OS.

Update: 2007-04-13
Ok there is some point to recording an appointment in UTC +/- offset for a groupware application used by a global organization. This was still a debacle and when MS tech support can't tell you what the state of your appointments is going to be after applying their provided fixes, then the overall point - they suck - stands.

2007-02-20T11:08:00.000-08:00

Stoopid MS Tricks part one:

2007-02-17T22:57:00.000-08:00

Looking for Your Wallet Where the Light is Good

There's a joke about a drunk on his hands and knees under a street light, looking for his wallet. Guy comes by, asks where he saw it last.

"Over there in that alley."

"Why are you looking for it here?"

"The light is better."

I had a situation where that makes sense. It was the case of a Weird Cisco Malloc solved (by someone else).

We'd been having malloc errors that shut down one interface on a Cisco GSR 12008 router. Very intermittent, and started the day before Cisco announced three vulnerabilities. Also, it hit another router in the district (but did not affect two others). I found a lot of things we could do to protect the router, including recieve ACLs and requiring a TTL of 255 for BGP traffic to ensure packets actually came from a neighboring router. Didn't help. I was going nuts trying to find a way to dig out information from the linecard, including memory and cpu status. Never did find anything. I was on the point of borrowing a line card from someone else when a coworker pulled it to try blowing any dust off. Turns out there was some, but the main thing was a RAM module dangling loose. He seated the module and the problem went away.

Blows me away that two routers could fail from something like that at the same time...

Also blows me away that there would be no error messages or that the failure would be intermittent. And that line card status is so unobtainable. These things are computers in their own right. They should have status instrumentation.

The simplest troubleshooting algorhythm is sort of a problem of a binary search, divide the possibilities in half. I think I will consciously weight the expense of checking a problem vs. its likelihood. That is, in this case hardware still seems an unlikely explanation (two routers failing the same way? Experiencing the same rate of thermal creep? Not buying it!). But it's so cheap to check. If you have a spare line card. We didn't have a spare long haul GBIC but we could have looked at everything else very easily. In other words, we could have looked where the light was good rather than wade through the various contradictory and redundant Cisco MIBs trying to make the thing reveal its secrets.

2007-02-17T22:55:00.000-08:00

D-Link AirPlusG DWL-G630 is machine washable.

That's kind of cool. Might be necessary, what with all the pr0n you internet pervs are into.

2007-02-06T21:41:00.000-08:00

Minor Fix (?) for Snort Decoder

We deployed a sensor where it could observe traffic hitting the external interface of a border router. Every BGP packet trips an "Experimental TCP Options" alert.

Now, I could disable that with a config disable_tcpopt_experimental_alerts directive, but maybe I want to watch for really odd stuff? As far as BGP goes, RFC 2385 providing for putting an MD5 hash as a tcp option dates from August 1998, about as old as Snort itself. (http://www.ietf.org/rfc/rfc2385.txt) So is it still appropriate to treat that as experimental and alert on it? As far as I know it's standard for BGP.

I know I can suppress, too. I think it would be better not to work around it.

I changed decode.c as follows:

line 3923 in 2.7 current as of 2007-02-06
From:
case TCPOPT_MD5SIG:
experimental_option_found = 1;
code = OptLenValidate(option_ptr, end_ptr, len_ptr, TCPOLEN_MD5SIG,
&p->tcp_options[opt_count], &byte_skip);
break;

to:

case TCPOPT_MD5SIG:
experimental_option_found = 0;
code = OptLenValidate(option_ptr, end_ptr, len_ptr, TCPOLEN_MD5SIG,
&p->tcp_options[opt_count], &byte_skip);
break;

I tried it out and recompiled, seems to work. That is, it no longer alerted on tcp option 19 (MD5SIG) and did alert on other things. It's possible I broke the decoder alerts, but I doubt it.

I followed the link provided in decoder.c http://www.iana.org/assignments/tcp-parameters
to see if there was anything else I should disable alerting on. Mostly dodgy stuff, if you ask me. But BGP using MD5 to transmit passwords is not a new thing.

I'll update here if the Snort developers pimp slap me for breaking decoder.

2007-02-06T21:23:00.000-08:00

ISOI II was very cool

http://isotf.org/isoi2.html

It was a closed gathering of security researchers, ISP reps and LEOs to which I wangled an invite. There was much cool stuff presented, but not much of operational interest. That is, I only took away ideas that would consume about 3 weeks worth of uninterrupted project time (or about 3 years job time). But the social networking was vast. I spoke with a Distinguished Engineer at Comcast, the inventor of the spanning tree protocol (http://research.sun.com/people/mybio.php?uid=28941). Various folks from Spamhaus. Botnet researchers, the Internet Storm Center (isc.sans.org), and on and on. Matt Jonkman of Bleeding Edge Threats (formerly Bleeding Snort), too. So if I have some issues, I have some people to talk to.

Golly, it's fun to be around so many smart people.

The coolest actionable idea for me (very timely, too!) was from a guy at Qwest who turned me on to filtering on TTL < 255 for communication with border routers. The idea is, you are only exchanging BGP data with neighbors. Maybe SNMP and ssh (or, gulp, telnet) from a management station. Set up pass ACLs for the management stuff, and then drop anything else with a TTL under 255. So you KNOW that any traffic directed to your router (not transit traffic, obviously) comes from a neighbor.

They can spoof addresses (hard with tcp, harder with BGP passwords) but they can't spoof the TTL. So the only place they can attack is from a neighbor (or an ip address that seems to be from a management station - and that's hard)

It rekindled my enthusiasm for this stuff. I've asked for a legal memo either authorizing or not authorizing deployment of a honeypot/honeynet. If I get the ok, I'll deploy and provide a listening post. You may need such a memo yourself as this can fall under wiretap statutes. It would suck to be prosecuted for violating an attacker's privacy!

2007-01-10T21:10:00.000-08:00

Changing the Size of a VMware disk

I recently set up a VMware virtual machine using a raw image file of a production windows desktop. (Whew! That's a lot of jargon!) If you don't want to read the previous post, it was simply booting linux, running dd vs. the disk used by windows to create the image file. Then I created a blank vmware instance and edited the .vmdk file to point to my image file and to reflect the correct disk geometry.

It was very cool. But the windows disk was huge - 40 gigs, in two partitions. Most of the stuff on the second partition was data, stuff I'd want from either the linux or windows side. So I copied it to a network directory. Now I had a ~36 gig partition, mostly empty.

After running for a couple of weeks, I was confident I didn't need the original disk any more. So I dd'd it. dd if=/winderz.dd of=/dev/hdb overwrote the original with my amended copy.

Next, I used ntfsresize to shrink partition2 to 4 gigs, leaving me plenty of room to install other apps and windows-centric stuff. Ntfsresize is awesome, and easy to use, even if you go command line like I did.

./ntfsresize --info /dev/hdb5
./ntfsresize --no-action --size 4000M /dev/hdb5
./ntfsresize --size 4000M /dev/hdb5
fdisk /dev/hdb
fdisk -lu /dev/hdb

The tricky part for me was getting a new image file with the smaller partition. I used fdisk to get the info about starting and ending blocks for each partition. Then I used fdisk to delete the partition info for the logical partition where the data was, and the extended partition where the logical partition used to be. I then created a smaller extended partition, just enough for the smaller logical partition I wanted. Then I created the logical partition. The thing to remember is that fdisk doesn't delete data when it deletes partitions, just information about how the disk is set up. So the bytes were still all layed out where I left them.

Then I could dd if=/dev/hdb of=/hdb.image.dd count=<(size of desired disk/ 512) +1> bs=512

dd if=/dev/hdb of=/home/myname/winbox.dd count=16209585 bs=512

One thing that threw me for a long time was the fact that I expected fdisk to work on the image file the same way it does on a real one. Fdisk will whiine about needing to set up cylinders on hdb.image.dd - don't bother. I never managed it, and the vmware disk file will define that anyway.

I set up a new vmware machine, using custom, windows 2000 pro, ide, and defined a 0.1 size disk. Once I created the machine, I deleted the flat file and edited the vmdk file. I plugged in the cylinders, heads, etc. and the size of the disk in 512 byte blocks:

RW 16209585 FLAT "/home/myname/winbox.dd" 0

The windows VM booted, and after an ntfscheck (caused by ntfsresize) I was back in business with a reasonably sized vm.

2006-12-22T12:28:00.000-08:00

Installing The Sleuthkit on Ubuntu 6.06 (Dapper)

TSK (The Sleuth Kit) is a package of Unix-based computer forensics tools. My interest atm is mainly in robust data recovery.

Usually, it's a good idea to install packages of software if you are running Ubuntu, which I am. In this case, the Ubuntu TSK package is version 2.03, and 2.07 is current as of today. Lotsa bug fixes and a few features added. So I started to install from source. There are a couple of dependencies that apt would have handled, but alas...someday I'll have to learn to create packages so I can save people some trouble.

TSK requires afflib. Afflib requires zlib and libssl.

First, install zlib
download file
tar -xzvf
cd
./configure
make
sudo make install (sudo required to install files in system directories. I understand it's bad practice to configure and make as root)

Now libssl:
sudo apt-get install libssl-dev

Now afflib
This was a pain. Fortunately, it was an unnecessary pain. There's no package available for Ubuntu 6.06 LTS. Compiling from source doesn't work either. But you don't need to - TSK just needs the code available for it's own compile. Found the following:

"The problem is that AFFLIB (for the AFF image format) requires zlib and
openssl, both of which do not seem to be included with Ubuntu by
default. You will need to install those packages and libraries. Most
systems come with those libraries, but Ubuntu does not seem to (I went
through the same pain a couple of months ago setting a system up).

Also, someone else had issues compiling Kubuntu with the version of
AFFLIB that was included in tsk 2.04, so you should probably update the
AFFLIB with the latest version:

1. Download version 1.6.26
http://www.afflib.org/downloads/afflib-1.6.26.tar.gz

2. Untar it.

3. Remove the src/afflib directory from TSK.

4. Move the afflib-1.6.26 directory to src/afflib (be sure you name it
afflib and not afflib-1.6.26).

5. Compile TSK as normal. "

So...do that. To continue with afflib:

download file
download tsk file
tar -xzvf
tar -xzvf
cd /src
rm -rf afflib
cp ./afflib

Now for TSK:
no configure, just make
This will put all the finished tools in /bin, not somewhere in the system folders. You may want to link to /usr/local/bin or some other spot. I cp'd sleuthkit-2.07 /usr/local/sleuthkit-2.07, then ln -s /usr/local/sleuthkit-2.07 /usr/local/sleuthkit

The symlink is for the convenience of apps like Autopsy (see below) so they can refer to a generic location and not be tripped up by updated versions of sleuthkit.

Now for Autopsy:
Autopsy is the html gui for TSK. You can do wonderful command-line things with just TSK, but by all accounts, you want this piece to tie it all together.

You need to know where you have sleuthkit installed, because it will ask.

tar -xzvf
cd
make
answer any questions...

follow its directions

2006-12-21T17:09:00.000-08:00

Installing VMWare Tools

VMware tools help the vm console and the rest of your desktop coexist. They install better video drivers, for example, and allow cut&past between console and virtual machine. VMconsole alerts you when you don't have the tools installed, so it's angst-inducing even if you don't need them. But it's not clear that the simple right-click, install vmware tools is only the beginning. What that does is prep the cdrom and provide a psueo .iso for the tools installation.

Cdrom should be set to auto detect. This can only be done when the virtual machine is powered off.

Then, for a windows guest (vmware term for virtual machine) you need to be logged in with admin rights and either manually launch the setup routine from your virtualized cdrom or if autorun is set (which it shouldn't be!) that will handle it for you.

Not a big deal, but I found myself wondering whether anything was happening when, in fact, nothing was happening. I'd like to see the tools installation option on the console say, "You need to log in now on the guest and install it!"

2006-12-21T16:18:00.000-08:00

Running a VMware image from a dd file

I spent a little too much time on this project, but I think it was worth it because of a bunch of interesting lessons learned. Basically, I had a windows 2000 desktop set up that I'd been using for years, with many shortcuts and tools installed, etc. I wanted to run it as a virtual machine rather than start with a fresh install and add all that stuff back in.

Thanks to this blog post on TechRepublic, by Justin Fielding, I had the outline of the steps to set up a Vmware virtual machine using a dd disk image. I found that the data for cylinders, heads, and sectors printed on the back of the HD wasn't the same as what I could get via software. I used losetup to mount the dd file as a loopback device, and fdisk to query the device. Note, you can also use losetup to set up a loopback device, which you can then mount as a partition and pull files from. Cool!

sudo losetup /dev/loop/0 james-backup/hdimage.dd
sudo fdisk /dev/loop/0

The number of cylinders for this disk is set to 4865.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/loop/0: 40.0 GB, 40020664320 bytes
255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/loop/0p1 * 1 510 4096543+ 7 HPFS/NTFS
/dev/loop/0p2 511 4864 34973505 f W95 Ext'd (LBA)
/dev/loop/0p5 511 4864 34973473+ 7 HPFS/NTFS

I plugged this data into hdimage.vmdk (the VMware disk configuration file) as follows:

# Extent description
RW 78156225 FLAT "hdimage.dd" 0

(78156225 = 255 heads * 63 sectors * 4865 cylinders)

ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "4865"

This shows two primary partitions, as expected, and the extended partition.

I ran into a wrinkle where the image would boot but fail. The first series of problems went away when I mounted the Samba share correctly - with the uid of the user needing to access it. I was a little confused by the file permissions on the remote server, vs. the file permissions on the local mount point. Clarity descended when I specified the uid as in the example below. Anything in < > is specific to your situation.

sudo smbmount -o lfs,credentials=/home//.smbpasswd,uid=

Breakdown:
//server/share is the server and file share. Typically a Samba server will be set up to share out a user's home directory if you connect with that user's credentials.

/mountpoint specify where on the local file system you want the server file share to show up. You could create a directory in /mnt, for example, or in /home//samba Then you cd /home//samba, and the remote files are there for you.

-o - specifies that mount command options follow.

lfs - large file support. See below.

credentials - This allows you to specify a file containing the username and password used to connect to the Samba share. Slightly less risky than sticking them directly in /etc/fstab

uid - the local user id you want to own and control the mounted Samba share.

Once I got that all lined up, vmware could open and use all the file locks and temp files it needs. But I still had a problem: the vmware virtual machine would start, but bomb while it booted, leaving me with a core file and some stale lock files. I couldn't vmware.log showed a "caught signal 25" entry each time. According to the man page for signal, Signal 25 is

SIGXFSZ 25,25,31 Core File size limit exceeded (4.2 BSD)

File size limit? Hmmm. The file is on a server, being accessed through Samba on a client. I checked around. On the local machine, ulimit reports "unlimited". Same on the server. In fact, copying the large image up there in the first place through scp didn't give me any problems. My friend Jeff found a parameter for mounting the Samba share, lfs. That solved the signal 25 problem.

Final note: it would be better (faster) to use the iSCSI Enterprise Target to store the vmware image file. Going through a file server is going to be slower than using a SAN. iSCSI lets you get block level access to the remote disk, effectively turning your LAN into a big SCSI cable. I just don't have a server with any unallocated disk space at the moment, but I do have some Samba servers with an oversupply of storage.

2006-12-11T10:52:00.000-08:00

Agents - a bad idea

I'm guilty of blog echo here. I'm echoing a post on taosecurity that echos a post on Matasano. Sorry. Sort of.

For a while now I've felt the obvious drawback of adding software to a production device - anything adds complexity, which undermines reliability and security. Whether that additional software adds enough in return is the question.

Matasano examines this credo in detail, with empirical evidence supporting the idea that you need to minimize agents. Agents are used for antivirus, desktop inventory and configuration management (especially for security functions like patch management, firewall and host-based IDS).

Read the Matasano post. Among other problems with agents:
1) Vendors are still in early-mid 1990's mode as far as responding to vulnerability reports. That is, they ignore them.

2) Agents are complex and invasive processes, creating a massive attack surface. This is not a theoretical problem; there's an extensive history of actual vulnerabilities in agent-based tools.

3) This inviting attack surface is present across an enterprise, typically. That is, there's a huge monoculture target waiting. The biodiversity analogy suffers from the same weakness all argument by analogy does, but it's still useful: where there's no genetic diversity, a population, if vulnerabile, is universally vulnerable to an infectious agent.

4) Further, most agents report to centralized servers, which themselves present inviting targets. Own that server, own the enterprise. Yikes.

So what's the alternative? Software vendors develop agents to control and collect data on otherwise unmanagable numbers of machines. There may not be built in mechanisms externally available to take care of this stuff. I think, for the most part, these mechanisms are present in modern OS's. Essentially, the agent and its problems are already provided, so don't add more. Also, I think the pull method, where an agent periodically checks in and requests updates and reports status, is more robust than the push method, where a central server issues directives. In both cases, owning the server is tantamount to control, but there are some beneficial corner cases for the pull method. In the pull method, you could still provide a malicious update on the server that the client's agent would pull down and act on in good faith. But that's a little trickier than the Stalinist, push method. Also, a pull method doesn't create a new, listening service that can be attacked without controlling the server.

I believe cfengine and puppet both follow this approach, but I haven't used either.

Of course, the element of authoritarian direct control is part of the appeal of these systems. So push will probably win over pull due to its superficial appeal.

2006-11-29T10:27:00.000-08:00

VLAN tag problem with OpenBSD tcpdump

Looks like the tcpdump/libpcap provided with OpenBSD has a problem with VLAN tagged traffic. The fix is to download both from tcpdump and compile fresh.

Note: usually you WANT to use the OpenBSD provided packages and ports, because someone more knowledgible than you tweaked them to work. Not in this case, though.

http://www.vorant.com/nsmwiki/index.php?title=OS_Anomalies

From #snort-gui:

(05:58:12) jontow: uh.. excellent? openbsd 4.0's tcpdump doesn't support vlan tags?
(05:58:33) jontow: ^:root@volvere:/nsm/manual-pcap# tcpdump -n -v -i xl1 vlan
(05:58:33) jontow: tcpdump: WARNING: xl1: no IPv4 address assigned
(05:58:33) jontow: tcpdump: syntax error
(06:15:07) jontow: yeah, what the shit.. if i compile it from source, too, from tcpdump.org the vlan filter still doesn't work
(06:15:08) jontow: grrr
(06:15:49) jontow: this sensor is useless without it
(06:16:03) jontow: stupid span port on these foundry switches send all packets still-tagged
(06:16:30) jontow: maybe they compiled pcap without vlan support.. :/
(06:34:47) rwatson: jontow: "doesn't work" in what sense? there are a lot of bugs in various IP stacks relating to mixing and matching things like promiscuous mode, hardware assisted vlan tagging, etc.
(06:34:56) rwatson: jontow: if you can't get it to work still, try disabling hardware vlan assist
(06:35:33) rwatson: jontow: this won't help with rule syntax errors, of course. :-)
(06:42:44) helevius: jontow: try naming a vlan to watch, like 'vlan 10'
(06:42:56) helevius: That might make a difference, might not
(06:45:08) helevius: It probably won't
(06:47:05) jontow: it literally is just the syntax
(06:47:18) jontow: it works fine, i've compiled libpcap/tcpdump from the tcpdump.org site and its fine now
(06:47:54) jontow: the bundled one just doesn't include support for vlan tags at all
(06:48:32) jontow: really odd that they didn't do that though.. oversight maybe?
(06:48:49) drape: i know on fbsd some interfaces don't support vlan tags :/. found that out the hard way.
(06:49:00) jontow: thats not a problem -- these do, and its confirmed on freebsd ;)
(06:49:38) jontow: richard; specifying a tag doesn't change the syntax error problem btw :/
(06:49:51) jontow: this means though; that i need to be recompiling snort
(06:49:58) jontow: since i'm sure its linked into the native pcap

2006-11-13T17:21:00.000-08:00

iSCSI targets in Ubuntu

No real contribution here: I just followed the directions on the Ubuntu forums

Then I went to the Microsoft web site and searched for 'iscsi driver' and found this link.

Install the software found there or at whatever updated link you find.

By default this will stick the software in your programs menu. (Start button, programs, Microsoft iSCSI Initiator) Read the readme file. It takes you through a couple of steps to test the connection and then make it. In Device Manager, I can see the MS iSCSI Initiator under SCSI and RAID Controllers, and an 'IET Virtual-Disk SCSI Disk Device' under disk drives.
I don't see how to partition and format. The Disk Manager had a problem, because I disabled the Logical Disk Manager services.

Ah, all is well now that I enabled those services. Disk Manager shows the disk, and I can create a partition and format it, just as if it were a local disk.

Just as advertised.

2006-11-13T14:19:00.000-08:00

Snort as an IPS

Someone in the local Snort user group asked me,

"Dear James,

Was wondering if you know of anyone who has used snort for line-rate
deep inspection on networks up to about 400 mb data flow.
And if so on what hardware. "

I replied,

"I'll ask around.

I don't have direct experience with this, but I'll share my thoughts anyway.

I don't use it inline for a couple of reasons.

1) separation of policy audit from policy enforcement (conceptual thing)
2) complexity in a firewall is a bad thing imho (luddite thing) There's a history of vulnerabilities in anything that does protocol decodes, because they are tricky.
3) fail-open and default pass less desirable than default block on my stateful firewall

There are certainly reasonable claims to be made in favor.

I believe Sourcefire claims they comfortably handle gigabit rates for their branded Snort appliances. I'm sure you can't use a lot of any -> any rules, though. Some tuning required.

If I were to build such a box, I'd get a Xeon 5100 series, with a 1333 mhz bus, and intel I/OAT compatible cards/chipset to offload some of the tcp/ip overhead. I don't *think* the disk is critical so you could go with a SATA2 drive.

http://www.siliconmechanics.com/i5917/Xeon-Server.php

Sourcefire uses commodity hardware, and thinks they will be able to use Moore's Law to surpass ASIC-based custom hardware. The basic proposition is, can Tippingpoint do R&D and manufacturing better than Intel et. al. ? I talked to a smart booth weasel for Tippingpoint at an expo (the only IPS weasel who knew what he was talking about in a field of 4-5 vendors) who thought they could, with FPGAs.

Since you have to have a higher confidence level in the blocking rules than in alerting rules, you might want to run a separate IDS. You can have a larger ruleset without DoS'ing yourself on a device that simply observes. Any problems it has don't affect the net."

I don't know if I was clear but he can always ask. Some points for clarification: IPS doesn't just watch the bad packets go by. It blocks what it identifies. The problem is, that identification must be more certain than an i.d. that simply generates an alert. Another problem is that behavioral/heuristic analysis will unpredictably alter network performance. The Visible Ops/ITPI folks would scream. Apart from network neat-freaks (to whose ranks I aspire to join), there's a real issue with allowing stimulous of an attacker's choice to alter the behavior of your network defenses. And, inevitably, some fraction of what an IPS blocks is going to be legitimate traffic.

The point about the additional complexity of an IPS is that a dumb, robust stateful firewall is going to be reliable. An IPS does more, has more code, more functions, more flaws. This is mostly, but not entirely, theoretical. Snort has had issues. BlackIce has had issues. I imagine the others have, as well. Ethereal (now Wireshark) and tcpdump have had issues, and these tools do things that an IPS must do. Protocol decodes must either attract really stupid programmers, or be really hard to do. I'm guessing the latter.

The point about the default block is that usually an IPS passes whatever it doesn't recognize as bad. Marcus Ranum explains why "Enumerating Badness" is a broken model. You want your firewall to permit only specific things, and block everything else. So an IPS must not be your only line of defense.

2006-11-03T13:07:00.000-08:00

Sometimes I am a dumbass

I made the following post to nagios-users. Note the 50-50 success rate in redacting the snmp community string:

James Affeld wrote:
> When I run just about any kind of SNMP check, I get suitable info at the beginning of the response, but a bunch of junk at the end. It even runs into the next command line. As you might gather, this is done via an ssh connection, using putty.
>
> root@silmec2:/usr/local/nagios/libexec# ./check_snmp -H 10.139.7.1 -o .1.3.6.1.4.1.9.9.48.1.1.1.6.1 -C REDACTED -P 1
> SNMP OK - 17448296 | iso.3.6.1.4.1.9.9.48.1.1.1.6.1=17448296Üjú·ôB¿¹ÈÈÜjú·ÔB¿`B¿¨B¿¢é·;;;;
> root@silmec2:/usr/local/nagios/libexec# PuTTYPuTTYPuTTYPuTTYPuTTY
>
> If I do an snmpwalk vs. that OID, it returns cleanly.
>
> root@silmec2:/usr/local/nagios/libexec# snmpwalk 10.139.7.1 iso.3.6.1.4.1.9.9.48.1.1.1.6.1 -c 55CC0000 -v 1
> SNMPv2-SMI::enterprises.9.9.48.1.1.1.6.1 = Gauge32: 17390392
> root@silmec2:/usr/local/nagios/libexec#
>
>
> I run into the same problem checking a Cisco Catalyst 6009 and HP 2524 switches. Any clues?
>

This string has been retired. I should have been, as well.

2006-10-17T17:28:00.000-07:00

Snort Rule Clinic

I gave a Snort rule clinic this summer. Slides in pdf are here and in Open Office Impress form here . I believe Bleeding Snort is going to post them on their site as well.

Feel free to adapt and correct. I'd like to know if you find any errors, but no attribution is necessary. My understanding of the GPL is that I was required to apply it to the presentation since I included Snort Community rules as examples, and they are released under the GPL.

I ran into some content-free slides recently, which irked me. I know that graphically, these slides are horrorible, but they stand alone pretty well for content. They aren't just an outline. I hope to carve the audio of the clinic up into 10 minute sections and podcast it, but I don't think that is crucial. (I also need an editing suite with a macro to auto-delete "uh"s.)

Really, the best source is the Snort documentation, which is the clearest I've seen for any software. Sometimes a second look from another POV is helpful, though, and there are one or two points I clarified. I also focus on the most important rule features, as measured by frequency of use in the rule set.

2006-10-17T11:31:00.000-07:00

Unsent Bugtraq Rant re: Web Vulnerability Checking

Someone asked Bugtraq about whether students in a security class should notify websites of vulnerabilities they have found. Most of the reaction was smarmy, bureaucrat CISSP types saying, "This is totally unethical!" I am sensitive to the problem of signal/noise on Bugtraq and decided against adding my own volume to it, but I did compose the following rant:

I will agree that this is probably the legal position; in Britain Daniel Cuthbert found to his regret that any interaction with a web server at all is fraught with peril. I think the legal position is pretty retarded, though.

Inspecting web sites for XSS is as valid and as ethical as fuzzing binaries. Get something straight: THE BAD GUYS ARE DOING THIS. They are no longer waiting for patches to reverse-engineer, if they ever were. Discovering vulnerabilities and disclosing them to vendors is a good thing.

Without disclosure, consumers are at the mercy of marketing weasels who value the perception of security much more than they value the reality, and FAR more than they value your well-being.

Even if it were not an indisputably Good Thing, you still have to construct an incoherent theory of trespass to craft a law disallowing certain types of interaction with a web server on the internet. Why did you set a web server up if you don't want interaction? Daniel Cuthbert simply added ../../../ to a URL. Web servers function by accepting URLs. That's what they do.

I'm not arguing for full-blown, unauthorized pen-tests. Nor am I arguing for the right to exploit vulnerabilities once found. I'm also trying really hard to stay away from argument by analogy. "Killing the messenger" is not an analogy; it's a precise description of the situation.

2006-10-16T21:31:00.000-07:00

3 Book Reviews: Ubuntu Titles

Moving to Ubuntu - Marcel Gagne

The Official Ubuntu Book - Benjamin Mako et. al.

Ubuntu Unleashed - Andrew Hudson, Paul Hudson

I received 3 Ubuntu titles and thought it might be useful to compare them. Ubuntu is a fairly recent Linux distribution that strives to be usable out of the box, with strong support. It has deep pockets and a thriving community behind it. You can get a free, fully-functional installation and livecd just for asking, or downloading. I admire a lot of the design choices that went into Ubuntu, such as limiting the use of the all-powerful root account, which can get people into trouble. The bare-bones server install is the cleanest Linux server I've seen - *no* open ports, minimal services. Just enough to log in at a console and then install what you want. On the other hand, if you want a LAMP server (Linux, Apache, MySQL, and PHP - the most popular combination on the internet), that's a one button install! Brilliant!

The only thing I don't like is the iptables firewall. A "linux for everyone" needs an easier firewall to deal with. (I love pf, written for OpenBSD and now showing up on other systems.)

I think all three books are pretty good, and your choice will depend on your technical level and religious ferver. If you are uncomfortable with computers, I think _Moving to Ubuntu_ is your best choice. If you are somewhat comfortable and into the philosophy behind Ubuntu, _the Official Ubuntu book_ is your best choice. If you are unintimidated by the topic, _Ubuntu Unleashed_ has the most detailed technical coverage.

Moving to Ubuntu - Marcel Gagne

This is the most approachable of the three books. Gagne has an enthusiastic, conversational, even narrative approach to the material. The audience is people stuck using Windows desktops because they don't know any Linux nerds willing to help them. I think it's a terrific book, and it showed me some cool things to do on the desktop. I use Linux mainly for servers.

It covers productivity apps very well. One quibble: he introduces GAIM, for chatting on various systems, and then introduces another tool for IRC, which GAIM handles just fine. The multimedia coverage is the best of the three books. The section on games is good as well, and I like his approach of getting a teenage nephew to recommend the best Linux games.

Like Ubuntu Unleashed, this book has a lot of material lifted from earlier works. I don't think that's a bad thing if the material lifted is generic. In this case, Gagne uses material from the slightly earlier _Moving to Linux_, which mostly used on one (non-Ubuntu) distro and mentioned some differences. Unlike _Ubuntu Unleashed_, the material was applied carefully. They even updated some things that didn't have to be, like an illustration in _MTL_ that had a graphic with a logo reading, "Welcome to Linux". In _MTU_ they cared enough to change it to "Welcome to Ubuntu". The chapters on Open Office are the same - and that's appropriate because Open Office IS the same. The GIMP is the same. So I think it's appropriate for the chapters to be the same.

Gagne pays some attention to the Ubuntu community ethos, but he's mostly concerned with showing someone unfamiliar with the system how to do the things they are most likely to want to do.

A good book, GREAT for newbies.

The Official Ubuntu Book - Benjamin Mako Hill, et. al.

This is at a midpoint in complexity. It is the strongest of the three in describing Ubuntu the phenomenon, rather than Ubuntu the tool. They honor their antecedents (especially the Debian distribution on which Ubuntu is built) and support projects built off of an Ubuntu base. The committment to the Open Source/Free Software community is very strong: even the book is Open Source, meaning you can copy, improve, and distribute it! Good technical details, few editing mistakes.

One area where this exceeds even Ubuntu Unleased in technical detail is in the future of the server side. While not yet ready, there are features that will make Ubuntu more suitable for server farms and clusters than it currently is. They also describe high end features like support for Red Hat's Cluster suite. Ubuntu Unleashed doesn't mention that, even though it is a retailored version of Fedora Unleased.

There are good points and advice throughout, and I picked up some neat tricks and tools. For example, I hadn't heard about zcat, zgrep, and zless, which work on gzipped files without requiring you to unzip them. Cool!

In the installation section, they include some useful tips like how to switch to another console in case you need to do something in the middle of the install. (I had to do that last week.) There's great information on setting up partitions, including one tip to separate /var/spool and /var/log because both can fill up if there's a glitch of some kind. I've long put /var on a separate partition, but that's an additional level I may adopt.

KDE is another desktop environment (Gnome is the default). TOUB gives the KDE flavor of Ubuntu, Kubuntu, full and fair treatment. Ubuntu Unleashed crams in a little Kubuntu stuff here and there.

I liked the treatment of bug reports in Chapter 6. That's the most realistic way the average user can make a contribution - catching and describing bugs in a useful way.

The discussion of scheduling jobs through cron was very good. I learned some stuff I hadn't heard before, such as using lists and ranges of times.

A couple of issues:

There is some very bad password advice on page 40, where the authors essentially suggest running a dictionary word through a 'leet-speak' filter, turning something like 'password' into p455w0rd' (substituting 4 for A, 5 for S, 0 for O). The bad guys crack this easily.

The discussion of the X-windows client and server on page 53 probably only makes sense to those who already understand what's going on.

The troubleshooting section for hardware was a little weak. "Want to watch DVDs? Check the forums." "Want to install a Tivo-like package? Check the forums." The book does a good job of describing the approach to software licenses and the exclusion of packages that aren't 100% free. But it doesn't do such a good job of how an individual can add those parts after the install. For example, playing dvd movies requires some additional libraries and the book doesn't provide much guidance. (Google "decss ubuntu" for starters)

I mentioned the editing is pretty good, no huge glitches. The chapter subtopic is wrong on 319-329: an earlier topic got stuck, I guess.

In sum, a good book and a great introduction to the Ubuntu community. Get this book if you want a family as much as an operating system.

Ubuntu Unleashed - Andrew Hudson, Paul Hudson
This is the most detailed of the three titles. It's aimed at a more technically proficient audience than the other two. It has the highest page count, and there's more print on the page. It also has the most demanding writing style; the other two are more conversational (especially MTU). It's perfectly clear, but the tone is not as reassuring to newbies. I think a Linux newbie who was fairly technical would still be comfortable, and it presupposes very little knowledge. It's mostly a matter of tone.

Part I, Installation and Configuration is about 260 pages.

Part II, System Administration, is about 170 pages.

Part III, Ubuntu as a server, is about 175 pages. It introduces Apache, Postfix, and other services.

Part IV, Programming, introduces Perl, Python, PHP, and some tools to use with C/C++ (but nothing on those languages themselves)

Part V, Housekeeping revists and amplifies Part II.

The good: I really like the organization. The other books are laid out for someone who has just installed, or is about to install,Ubuntu. This one expects that you will read much more material before hitting the keyboard. That's not a bad thing, for its audience. Home users won't have a lot of use for the mass deployment advice, for example, but IT folks might. In particular, this is the only book of the three to cover using Kickstart to automate installation.

I like how it gives two tables of contents, one brief, one detailed. (The detailed table of contents is 23 pages! The index is 62, but misses some keywords.) Each chapter recaps important commands, and provides links for further information. That's a great template I wish other books would adopt.

It briefly covers the history of Linux and the Ubuntu distribution of Linux. The other two books are a bit more evangelical about Ubuntu. This intro is more for the "Just the Facts, Ma'am" crowd. It is enthusiastic about Open Source and Linux in general.

They lift material from other books, especially Fedora Core Unleashed. That's not a bad thing - is it necessary come up with a new way to describe how TCP/IP works for each book you write? There's a lot of generic information that applies to most distros. A book that was only about Ubuntu and not general system administration would be pretty weak, in my opinion. This has a lot of good information about
running your system.

It introduces servers like the Postfix email server, and programming languages like Perl. Huge books are written on some of these topics, so you might wonder whether there was any point in a short chapter on them. In most cases, I'd say there is some point. You may not master Perl with what they give you, but you might be able to figure out some things. The Sguid proxy server treatment is short but could be very useful.

The Bad: The book was hastily thrown together, lifting or adapting a lot of material from Fedora Core Unleashed. As I said earlier, the repetition of material is not bad if the material is generic. But I don't expect to hear a lot about obtaining RPMs (software packages used by Fedora, among many other distros, but NOT by Ubuntu) in an Ubuntu book. At one point, they actually refer to Ubuntu Core! I'd lay odds there was a search and replace function used to swap "Ubuntu" for "Fedora" when it should have been for "Fedora Core." At another, they refer to different backup applications being available to the "business oriented" version. Ubuntu doesn't segment itself this way - that's a Redhat characteristic. There are tools mentioned that just aren't part of Ubuntu, as well.

The Security Chapter is terrible. It even describes itself as "all you need", but it isn't remotely enough. I give the other two books a pass because they are mostly aimed at users, not system administrators. This book needs a third coauthor who is well-versed in locking down internet-facing linux boxes.

A minor thing - the book is riddled with examples that will be dated before it goes out of print: hard drive prices and capacities, etc.

Chapter Notes - lots of nit-picks. Like I said, this is a good book overall.
Chapter One: Good short history, covers the bases of why this Matters.
Chapter Two: Good hardware compatibility resources. Useful USB incompatibility warning. Partition info is good - hint on partition for laptop suspend.
Chapter Three: it's customary when providing an example password to advise against using the example on your own systems - the examples often wind up in password cracking databases.
Chapter Four: Post-Install Config. Good tip on making a backup copy of each
Chapter Five: really nice description of the file system layout.
Chapter Six: X-Windows. It misses a chance to show how to set up a remote session. This is handled elsewhere, but why not here?
Chapter Seven: Software. This chapter is missing a section on dpkg, which is the underlying package management tool used by the other tools they talk about. They discuss it much later.
Chapter 8: Browsing/Email - good discussion of using mail from the command line (useful for scripts!)
Chapter 9: Productivity. Glosses over the use of Open Office, which is probably o.k. The audience can figure that out or get an Open Office book. Overstates coverage of groupware in Chapter 8 - it wasn't "in detail". Good call to plug Codeweavers for running native MS Office. Some people have to...
Chapter 10: Multimedia. This chapter suffers most from its origins in Fedora Unleashed. It's riddled with references to RPMs. Installing a t.v. card requires editing kernel modules, should refer to Chapter 35.
Chapter 11: Image Manipulation. VERY BAD ADVICE telling people to enable remote X sessions by entering xhost+ This is unfathomable!
Chapter 12: Printing. Pretty much just a UI run-through.
Chapter 13: Games. Good coverage, nice to know that the main first person shooters are available natively. Also good to know about Cedega, a games-oriented emulation package. Another Redhat-ism: Cedega is "not available via Yum"
Chapter 14: Users. Good coverage of user accounts, including those used by system services. Learned some stuff. User disk space quotas are mentioned, but I found the discussion unclear. User Accounting is a useful tool for security as well as old-fashioned timeshare billing, and they cover it pretty well. Really odd advice that you can edit /etc/shadow with a text editor. This is unsound.
Chapter 15: Automating Tasks. This was great. I learned new stuff, like scheduling jobs for a list or range of times. I liked the shell script introduction. There's an odd reference to Tripwire and Logwatch being included. You can install them, but they aren't included by default. Maybe another Fedora leakage? I liked the shell
script examples, but would have liked to see a few more, especially for the If clauses.
Chapter 16: System Monitoring. I learned good stuff about Top, time, and watch. Vmstat is a new one to me.
Chapter 17: Backups. It's arguably out of scope, but I think that they should mention that these days backup tapes have to be handled like evidence, with a chain of custody and logged distruction/wiping/disposal. Really liked the coverage of tar,
learned about incremental backups with it. dd coverage good, especially the warning about confusing source and target. Odd discussion of KDE gui backup tools - "archive has...function of system administrator...no GUI necessary" This applies equally to the Gnome tools like File Roller. Why say this in section on KDE tools? Liked the mc tool, but there is no package available for Ubuntu. They refer to obsolete rcp command, and say they previously mentioned it. I don't think they did! The index doesn't cover it. Big confusion in ext2/ext3 undelete: you can't undelete in
ext3 file system! ext2fs doesn't have the information available to do that. They need to make that VERY CLEAR, rather than misinforming you about undeleting. Good recovery information. GRUB boot floppy -great idea.
Chapter 18. Networking. Good explanation of purpose and use of loopback. Chapter needs editing. Private IP space handled twice, once incorrectly. They provide a good link to info on wifi adapters. Subnetting is pretty muddled, but should't hurt anyone who actually has to do it. The discussion of fiber optic cable is oddly wrong in one section, after getting it right in a previous section. It is not a 100 mbs media. It's 10mbps, 100mbps, 1 gig, 10 gig...depends on fiber characteristics, length, and the devices attached.
Chapter 20: Web Servers. Is C really one of the most popular CGI languages? I'd think PHP, C#, VN, Python...
Chapter 21: Nice comparison of MySQL vs. Postgres, and what type of tables you need in MySQL to get certain features.
Chapter 22: Good NFS intro, I think. Samba references really ought to include "The Official Samba How-to and Reference Guide" (TOSHARG). The section describes stand alone file servers, but you can set up a full-fledged NT 4.0 style domain with Samba. You should at least mention it and give a pointer to more info.
Chapter 23: ftp. Good emphasis on insecurity of this clear-text protocol.
Chapter 24: email. Typo/brain cramp in Qmail section: "Postfix is designed to be easier..." but we're reading about Qmail. Exim is claimed to be more secure than either Postfix or Qmail, but Qmail has an unclaimed bounty for any security related bugs. Word choice: "Postfix is...recommended client." An MTA is a client? Hmm. Good overview of Exchange alternatives.
Chapter 25: Proxy. Good examples for using access control lists in Squid to limit connections by source/destination and content. Should be useful to get up to speed.
Chapter 27: Perl is a backronym. The name was chosen and only later turned into an acronym. There are two, equally correct and they should have mentioned "Pathologically Eclectic Rubbish Lister" Lots of great Perl books, some as big as this one. This serves as a quick intro and maybe refresher.
Chapter 30: C/C++. Doesn't address language, unlike previous chapters on Python, PHP, etc. Just tools. Debuggers, development environments, etc.
Chapter 31: Securing the Host. Most disappointing chapter in the book. And there's even a claim that all you need is in this section. While Nessus may not identify weaknesses based on patch level as the book claims, it can do deeper tests and see if a vulnerability is really present. Nmap is NOT a substitute. For firewall configs, it claims that gnome-lokkit (for GUI) and lokkit (for cli) are included. They aren't and don't appear to be part of the Ubuntu repository. This looks like another Redhat-ism. There's nothing on mounting file systems with suid limitations, using a central log server so you can trust the logs, exporting the tripwire database so an intruder can't just register the trojanned binaries. There's so much you can and
should do, and it is not covered at all. CERT has checklists, SANS has checklists, there's a wealth of information out there that they could point us to if they aren't up to covering it themselves. I'd scrap the whole chapter and start fresh in another edition.
Chapter 32: Tuning. This has good info on hard drive tuning. Major clue - disable atime for partitions seeing many writes/second. Each atime update is another write.
Chapter 33: Shell Master Class. Revisits the command line, really good information. I learned a lot about the less command, and screen.
Chapter 34: Advanced Apt. Good coverage of (finally) dpkg and aptitude. I'd call it complete.
Chapter 35: Kernel maintenance. Good info on kernel modules. I can't swear the kernel patching stuff is good (no direct experience) but it looks complete.
Appendix A: references are good. Mailing lists, chat rooms, websites. Seems complete.

2006-10-12T16:41:00.000-07:00

Tweaking IPS/IDS Vendors at Trade Shows

If you follow the NSM (Network Security Monitoring) philosophy, you focus on collecting and analyzing 4 types of data: statistical, flow/session data, full-content packet capture data, and alert data from an IDS (and possibly other sources of alert data). The best place to learn about the approach is here:

Snort is my choice for an IDS because it is Open Source (source code auditable, available for outside improvement especially feature development), which means it's Free-as-in-Beer as well as Free-as-in-Speech. Got a rant on the subject. It is incredibly flexible: I can read every signature and understand what it is trying to detect; I can get the output in any number of formats, which means the data winds up where I can use it best. Some vendors want you to buy a console that will cost as much as the sensor!

Sguil is my choice for viewing NSM data because it collects all the data I routinely need to look at and sticks it on a panel where I can use it. I've heard several analysts say sguil makes them four times as productive, with greater detection success. I won't go into all of why this works so well, but the big win is the database. Sguil stores data in a mysql database whose schema scales much better than other, similar projects. Here's an example of how it works: suppose you have an alert from your IDS. The state of the technology being what it is, you aren't sure if this is a real attack or not. By right-clicking on the source of the alert, you get a menu that offers options to see what other alerts this source has been involved with. If you see multiple attack signatures triggered, that's pretty good correlation and confirmation. You can also click a radio button to look up information about the owner of the network that source is from. You can click a button to display the rule that matched the traffic and triggered the alert. You can see the packet payload with another click. And - and this is the killer - you can right-click on the event id and either get a plain-text transcript of the connection, or get a bit-for-bit copy of the communication between source and destination. You can see the whole thing, as it transpired. You can send it to other, potentially wiser people for their analysis. You can replay the traffic to test solutions and defenses. You can extract any attack tools downloaded. Finally, if you confirmed that it was an attack, and the transcript showed it succeeded, you can easily bring up a record of all connections between the attacker and other hosts on your network. Even better, you can look at all the susequent network traffic of the compromised destination. It's very, very cool.

Back to the narrative.

I went to a local security expo where four vendors of IPS/IDS products had booths. They make what are essentially tricked-out firewalls. The most crude form of firewall, the packet filter, can make block/permit decisions based only on surface aspects of a single packet, like the source or destination IP address and/or port. Each packet is examined in isolation, which is not good.
A stateful firewall keeps a table of connections, so it can examine a packet in relation to other packets in the same conversation. For example, unlike a packet filter, a stateful firewall can look at a response packet and know that it is really in response to traffic initiated by a host allowed to have that kind of traffic. Stateful firewalls are still limited to surface aspects, but it can look at them in context, which is a big improvement. Application firewalls (Sometimes called "Layer 7") actually model/decode application level protocols. For example, one might be used to discriminate between legitimate HTTP traffic (web traffic), which usually travels to a web server on TCP port 80, and a P2P (Peer-peer) application like the old Napster, which might sneak out on TCP port 80 pretending to be web traffic. A half-step beyond the application level firewall is the IPS. They look deeper into the packet than an application firewall might. They use IDS approaches to looking for attack patterns and behaviors, and go beyond issuing an alert. There are pros and cons to this, among them that the behavior of the firewall may change in response to stimulous of an attacker's choice. An attacker may be able to issue stimuluous that will cause Bad Things to happen, like make blocking decisions that prevent necessary communication. On the other hand, they can keep other Bad Things from happening, like preventing malicious web sites from taking advantage of specific weaknesses in web browsers.

On to the tweaking: I already mentioned why I like Snort and Sguil. I went around asking the booth weasles for the IPS vendors if I could view the signatures and write my own. What kind of output options did I have for reporting. Could I get Flow data? Full content packet captures? The reps for McAfee, ISS, and RealSecure did not cover themselves with glory. For the most part, they had no idea what I was talking about. I don't think they would have recognized a packet dump. I got much blank look and repitition of previous points. (Kinda reminded my of my 3 year old son: "I want ice cream." "You can't have ice cream right now because blah blah blah." Blank look. Shake head to clear it of irrelevant and incomprehensible bullshit. "I want ice cream.") They might as well have been selling refrigerators. "Now, some people don't know that Maytag models give you cancer." The Tipping Point booth weasel did know what he was talking about, and even more, he knew what *I* was talking about. He was pretty comfortable defending the approach his company took. Most people don't do real IDS stuff.

2006-10-09T14:41:00.000-07:00

Why You Shouldn't Work with Sharp Objects When You are Tired

The other day I was re-running some fiber optic patch cables on a rack in the server room. If you haven't worked with them, they are very thin, flexible, fragile. In many places, they are run through conduit even on a rack.

The reason for me rerunning them was that the person who did it originally wove them like it was basket-making class. Send one patch cord through this bundle of cables, that one around the bundle, and a third around the other way. Make sure this new bundle of fragile cables interferes with every bundle it crosses...

Ugh.

Anyway, after an hour* I get it all smooth, shipshape and Bristol-fashion. Not quite up to the standards exemplified here but pretty good. I am plugging the patch cords into the switch, when I deem that the patch cords need one more tie to keep them neatly bundled within 1 foot of the switch. I use a zip tie, and need to trim the excess. I get the snippers out.

I cut one of the fibers.

I cuss a bit. I know better than to leave a dead patch cord in the bundle, especially with a seemingly ok end on the other side. So I go to cut the dead one out of the bundle.

I cut another of the fibers.

I go home and fix it the next day. (No server outage - this was for new stuff.)

Lesson learned - watch for fatigue-brain. You can commit some unfathomable errors. If you can learn to recognize it, you can avoid being a Menace to Technology. Some tasks require my Good Brain, some are safe for Low Technology Days.

*This is a clue. It shouldn't have taken an hour, even with the other cleanup I did.

2006-10-09T11:48:00.000-07:00

Training Plan

I'm crafting a proposal for the fu-acquisition I expect to accomplish by the end of the year. I'm asking to go to USENIX LISA http://www.usenix.org/events/lisa06/

I went last year and it revitalized my interest in my career. It was really cool to walk into the room and lower the average I.Q. (Yeah, yeah, I should be used to it :P ) Seriously: this was awesome. Most of the instructors literally wrote the (O'Reilly) book. The presentations that make it a real conference (unlike SANS, which has trappings of a conference but is really just training) are amazing. I was torn last year about whether to attend class or a talk. Again, this year. So I want to go again. The line up is again stellar.

I also want to go to Shmoocon, again in Washington, D.C. It's a small, high-end security conference. Only, not stuffy. A little bit of hacker underground to spice things up, but mostly real researchers.

One thing I don't want is Microsoft Authorized Training. Gawd. I was sent to one about 8 years ago that sucked out loud. "To open a file, click the 'file menu' and select the 'open' option." This was for a desktop and network management product. If you think McDonalds sells food, then you might regard this homogonized, standardized crap training. And another thing: Windows 2003 server is designed so that even MCSEs can use it. Training is superfluous. It would be a better use of dept. budget to send me to a triple feature of "Saw III", "Texas Chainsaw Massacre - the Beginning", and "Grudge II" for five days running. Including popcorn. I think that would be a more cost-effective way to induce a similar level of mental illness.

I will grant that actually making that stuff *work* is fairly complicated. But that's a function of shoddy software, and running through the user interface for 5 days is not going to address any of the REAL operational headaches of using the awful stuff from Redmond. What fails, fails under the hood where you can't get at it.

Another thing I don't want: glorified vendor trade shows aimed at non-technical CIOs. I want to see math. Exploit code. Configuration files and options. No "magic security spray."

I do pretty well with text books and web pages, but sometimes it's good to mingle with smart people who have solved hard problems so I don't have to.

2006-10-04T18:52:00.000-07:00

Toorcon Report

This year I went to security/haxor conference Toorcon for the first time. I have attended USENIX LISA, which rocks, and SANS, which does good training in the guise of a conference. This was a little...different. Not as insane as Defcon, not as polished as I hear Black Hat is, it consisted of a few hundred self-identified members of the 'Digital Underground', mostly whitehats, but some definite criminals. (Note - don't take offense if you are a Black Hat. It's a crime. I'm not calling anybody a terrorist.)

Very serious presentations, even if some of them were hilarious. One was done remotely and anonymously. It was a little dry, but the technique works.

The Capture the Flag tournement drew three teams competing to hack servers in the tournament network. The Midnight Research Labs crew completely dominated the other two, which consisted of much less seasoned attackers. By dominated, I mean almost shut out. Hundreds of points to nil. Only toward the end of the second day did another team get any points at all, and one team LOST points for losing a server. I am acknowledging the MRL, not dissing the other two teams. Hell, I didn't even enter the damn thing and wouldn't have gotten far if I had. I just play defense.

A licensed private investigator who runs "the largest privately held investigation support service company in the country" gave a scary talk entitled, "Privacy is Dead: Get Over It."

Some points: You may one day lose insurance coverage if someone gets a list of your Amazon purchases and it includes "Recovering from H.I.V." Credit card companies track your purchases, and sell your profile to pretty much anyone. The Feds have outsourced profiling efforts to businesses like Choicepoint, so you can't get FOIA satisfaction. You can't make Choicepoint tell you what they do with your information. You can't even SEE their records of you. Subscribe to "Soldier of Fortune" ? That might get you on a list of suspects. It WILL get you on multiple marketing lists. Every place you have lived, all kinds of crap, is easily accessible. In a couple of hours at most, a fairly full dossier can be compiled for a background check or whatever, without any field work at all. This talk went an extra HOUR, twice the scheduled time, and rolled right through the lunch break. Hardly anybody left.

VOIP (Voice Over IP) vulnerabilities - this stuff isn't news, nor did the speaker claim they were. What he did was demonstrate some exploits, like retrieving voicemail by spoofing CallerID. Some cell phone service providers use nothing but CallerID to authenticate access to voicemail. Well, VOIP software and even hardware allows the user to set the CallerID to whatever s/he wants. Duh.

Bridging - you can spoof SIP (Session Initiation Protocol) packets and set up an unsolicited conference call between two people, who will each think that the other person called them. The speaker passes out cards showing how to do this to women he meets in bars.

He played some audio of these exploits in action, and tied it to a computer model of sound processing in the brain. The model had nothing to do with the topic, but was kind of cool. He had a 3-D application showing activity in the "brain" during playback.

I attended the "Deep Knowledge Seminars", which were basically just regular presentations, but an extra day of them you have to pay for. I gambled and registered early to avoid a steep price increase. The gamble was that the lineup and topics weren't announced....it turned out ok. I found one presentation kind of useless, but the rest were good, including one I planned to skip. That was a consistent refrain: the things I was inclined to dismiss were pretty good.

One highlight: Dan Kaminsky is not a serious researcher - he's too busy laughing his ass off to qualify as serious. But he finds very interesting and hilarious stuff. For example, when notorious criminal hacker Sony Corp. overrode user/owner action and installed a rootkit when someone inserted a Sony music CD (something that should have seen prosecution), Kaminsky analysed DNS traffic to track the scope and spread of the infection. He chucked his scheduled presentation and showed a truly sick hack that requires a little explanation. (Apologies if you already know this stuff) DNS is the service that translates a host name, like www.mywebsite.com, to an IP address a computer can use to connect to. It uses small packets, intentionally limited. (If you get a large enough DNS response, the protocol specifies using a different approach than usual.) A covert channel is where someone creates a communication link through a protocol not intended for that purpose. For example, AIM and Yahoo Messenger (et. al.) will try to use 'standard' ports to connect, but if they fail, will try to phone home using a port generally intended for web traffic. The reason is most networks allow web traffic, so you can use port 80 (assigned by the IANA(?) for HTTP, the web protocol). That's the simple model. More advanced covert channels will use the actual protocol, but take advantage of padding and the like to carry the secret information. Anyway, Kaminsky abused the DNS protocol and stuffed streaming video into a covert channel. Because video is large, and DNS packets are small (by default), this is the most extreme case I can imagine. Anyway, Dan is someone to watch. Very, very smart guy on a staggering array of topics.

It's not all serious stuff. There are massive parties and many of the participants regard drinking as a competitive event. After the Con, there were two trains north from San Diego. I was on the 8:20 PM train. I heard unconfirmed reports of drunk, large, hairy, naked guys from the con causing a ruckus on the 9:15 PM.

I'm pretty introverted and boring, and don't drink. So I didn't participate.

It was interesting to see folks I'd met elsewhere, some at LISA 2005.

I plan to go again. I'll have a new baby around then (if all goes well), but San Diego is a pretty awesome place to visit and I missed my family. So I'll try to bring them.

2006-09-27T15:21:00.000-07:00

Book Review: Linux(R) Quick Fix Notebook (Bruce Perens Open Source) by Peter Harrison

I'm a big fan of the cookbook approach to tech books. I usually don't have time to read a book to get a broad and general understanding of a topic. I'm usually after what this book promises: a quick fix. I want answers to discrete problems. That's what _Linux(R) Quick Fix Notebook_ delivers. When I did have time to read an entire chapter, I learned a lot. When I flipped to random pages, there was a good chance I learned something. It's full of gold nuggets and neat tricks.

I work in IT, and I often show someone something that, to me, is pretty basic. But it saves them a lot of time. This book addresses those gaps in my own knowledge: the basic stuff I never happened to pick up. Sometimes it's so basic nobody bothers to write about it. Until this book.

Unfortunately, I couldn't hold on to the review copy long enough to finish it. But I'm buying a copy for myself. That should tell you something! Especially when I have about a dozen books on linux and unix system administration already. This approach works for me, and this book implements that approach really well.

2006-09-27T15:15:00.000-07:00

Book Review: Tao of Network Security Monitoring

(From my review on Amazon)
This is a great book. With most geek books, I browse and grab what I need. With this one, I even read the apendices!

At first, the author's tone put me off. He spends the introductory chapters talking about the "Way" of Network Security Monitoring, (capitalized) and how it's much better than other approaches. It felt a little like, "My Burping Crane Kung-Fu will defeat your Shining Fist techniques!" I really didn't see much difference between what he was talking about and other approaches. I admit to being much newer to this discipline than the author, and he has an impressive appendix on the intellectual history of intrusion detection (uncapitalized). So it may be that the lessons he advocates have already been internalized; my exposure may have been to a field that has already moved up to his standard. But I have a hard time imagining that intrusion analysts have ever been satisfied with a single approach with no correlation. As I understand what he means by upper-case NSM, it's basically the efficient use of multiple techniques to detect intrusions. I can't see trying to argue the contrary position.

Ah, but then we get to the good stuff. He goes through the major types of indicators and the means of reviewing them. He covers the use of a number of important tools, but doesn't rehash what is better covered elsewhere. For example, he doesn't bother covering Snort, because there are plenty of books on Snort already. If you are reading the book, it's almost a certainty that you are familiar with Snort. Good call to skip over that. Instead, he covers some other tools that might be useful in the same area. He also refers to tons of other books. I made a lengthy wish-list based on his recommendations and they've been good. (He also reviews exhaustively here on Amazon). So this book is like the first stone in an avalanche- it triggers the acquisition of many other books.

The book provided many 'light bulb' moments. For example, he talks about giving up on source-based focus. In a world where a DDoS attack is currently using 23,000 separate bots, we may exhaust our resources tracking low-value drones. So focus on the targets they are after: light-bulb! In spite of my earlier resistance, I was soon going through it as eagerly as I did with the Patrick O'Brian Aubrey/Maturin novels. It's fun to read such clear, authoritative writing.

One quibble - he trashes the SANS intrusion detection course, which I took and thought was terrific. He has taught the class, and considered the course material out of date. Maybe they have updated, but his book didn't contradict anything in the course as I took it 1.5 years ago.

2006-09-27T14:41:00.000-07:00

XP SP2 Firewall is a joke!

(14:24:22) jimmythegeek: HAHAHAHAHA!!! OMFG!
(14:24:46) jimmythegeek: I installed XP sp2 on a workstation. Get a report that a terminal emulator is very slow
(14:25:03) jimmythegeek: takes like 15 minutes to connect to this old minicomputer, but it connects
(14:25:17) jimmythegeek: Fix is to add an exception in Windows Firewall for that port.
(14:25:41) jimmythegeek: WTF? "We don't know how to drop packets, but we can sure slow 'em down! It's called playing to your strengths."
(14:26:22) jimmythegeek: A firewall should stop traffic or permit it. Or rate-limit, if it is fancy. This is...just sad.
(14:27:05) WuTang: :D
(14:27:10) WuTang: that's awesome

2006-09-25T18:58:00.000-07:00

Book Review: Visible Ops Handbook, Kevin Behr, Gene Kim, and George Spafford

You know the old saw about, "Sorry for the long letter; I didn't have time to write you a short one." ? This is a short one, but I don't feel at all cheated on page count. It's a Good Thing when a book covers the topic...and stops.

The authors codify the operational approaches that highly proficient IT shops have adopted. I'm hostile to dumb performance metrics, but using some measurements even I can agree are useful, they identify high-performing organizations. Some of the metrics: unplanned downtime, Mean Time Between Failure (MTBF), Mean Time to Repair, % staff time spent on unscheduled|unplanned work, and ratio of servers to administrators supporting them. They noticed a quantum-gap, where the outfits that did well in these areas tended to do well in all of them, and there wasn't really a continuum. Organizations were either high performers or low, not a lot in the middle.

Turns out the high performers all independently adopted similar operational approaches, and there really isn't a middle way. There's a discipline to the discipline, and it starts with "the only acceptable number of unauthorized changes is zero."

High performing IT shops have a culture of change management. They cite a stat indicated that 80% of outages (incidents or time? both?) are self-inflicted. That's obviously the place to look for improvements. And you won't get improvements with just a little change management.

This approach has a lot to offer besides operational efficiency. IT goons have to deal with useless auditing and compliance directives. (Some of it is worthwhile, but it looks like even the worthwhile efforts are not well done in practice.) Having effectively managed controls in place makes for auditable networks.

The Mean Time to Repair is improved by a Culture of Causality. Once you have the change control in place, you can have faith that you know when something changed, and look in those places for the cause. Proficient shops - even those running Windows - reboot 1/10 as often as their less proficient counterparts. It's possible to hit a 90% first-fix rate.

They claim (and I believe) that it's also cheaper to rebuild than repair. Figuring out what went sour is time-consuming and uncertain. Automated rebuild is the way to go.

Interestingly, they claim that the frenzy of patching that many of us go through is not part of the culture at these proficient shops. A patch is a change, and subject to the same build verification any other architecture change would be. Consequently, OS patches get rolled out more or less organically, as part of a whole system. This is a little harder for me to swallow. I have been immersed in the SANS koolade. Depending on the application, I don't think you can wait for some patches. I do grok the "one, few, many" approach used by Tom Limoncelli (http://www.aw-bc.com/catalog/academic/product/0,1144,0201702711,00.html) Pilot a patch, test, expand to a pilot group, test, release to production, cross fingers. If you have a disciplined shop, this approach works. If you are like the rest of us, the balance of risk suggests to me you are better off dealing with the turmoil a patch might cause than live with a worm outbreak.

Another interesting point is that change management is MORE important during a crises. Convene your change approval team but stick to the discipline lest you make things worse.

The authors claim that a transition to the techniques and culture that the high performing organizations have in common has been done in a few months. I don't see it happening that quickly around here, but we could certainly get started. They outline 4 phases:

1) stabilize the patient - set up an "electric fence" so that you can monitor configuration changes and hold staff responsible for unauthorized changes. Confining changes to those approved by a change management team, and only during maintenance windows will have an immediate effect, they claim. But accountability is key - without it, there's an inevitable slip back to the sloppy practices everyone is used to. The fence comes from tools like Tripwire, which can tell you when things change. You can then refer to authorized changes/work orders and see if they match up. If not, some coaching is in order.

2) ID the fragile systems you don't dare touch

3) develop a repeatable build library so you can start moving services off the systems identified in #2

4) continuous improvement

I am not sure how much this applies to my environment - we are pretty stable, but not proficient. Most of the action on our net is at the desktop level, and I think this is aimed more at network operations and the data center than the Help Desk. It has me thinking, though. If we could extend the principles to everybody, what would change? What would it look like?

2006-09-25T11:52:00.000-07:00

Rant: arithmetic operations on ordinal numbers

In virtually every discussion of computer|network security and asset protection, people trot out a risk equation on the lines of:

Risk = Threat x Vulnerability x Cost

This seems brain dead to me. Risk is the expected monetary loss from an event. This is a little better:

Risk = (Impact of an Event) * (Probability of an event)

Let's look at these factors. The Impact can have a dollar value associated with it, which can be more or less successfully generated by looking at replacement cost, revenue loss, etc.

The other factor, Probability, is going to be one of two general levels of accuracy. In some cases, you can know the probability of an event is one (that is, certain). You can be certain that an unpatched Windows file server exposed to the internet will be violated, probably within 2 hours.
http://isc.sans.org/survivalhistory.php In all other cases, you are pulling a number of the air. Or out of your ass. I've actually read a web publication that claimed to assess earthquake frequency and felt it could do something with that data in a risk equation. I don't buy it. But on to the rant.

Usually, the Risk Equation is done with qualitative factors, for example, at

http://www.sans.org/reading_room/whitepapers/auditing/1204.php , in section 2.2.4 on page 4

The author describes "Qualitative Risk Defined Mathematically".

Relative Risk = Asset Value x Vulnerability x Threat

To the author's credit, there is no actual attempt at doing math. But I have seen (and, at gunpoint, participated in) security assessments where these factors are assigned numeric values. So for example, a file server might get a 4 on a scale of 1-5. A vulnerability guesstimate would be, oh, 3. (But again, that number is pulled out of the air or wherever. YOU DON'T KNOW how vulnerable an OS is. Is there a Zero-Day attack employed by the bad guys? You either are, or are not vulnerable. I don't know which is the case. And neither do you. The best you could do is a qualitative ranking based on history, which is of unmathematical accuracy when predicting future performance. This ranking could be useful in thinking about what platforms are used for which purposes, but it should revolve around the skill level required to successfully compromise the assset. For example, "This is unpatched - Vulnerability = 5. This is patched, but the OS has a monthly patch cycle so it's almost certain that holes exist which haven't been found by the good guys - Vulnerability = 4. This OS has had one remote root in the default install in 6 years, we'd have to posit an unknown vulnerability in the absence of any history of published exploits - Vulnerability = 1")

Where these things go sour is when you multiply rankings. ( Impact = 5 ) * (Vulnerability = 5) = (Risk = 25) BZZZT!!!!

Ichiro Suzuki had the most hits in Major League Baseball in 2004. Ranking = 1

He was (I'm making this up) the 5ooth tallest guy in the League (MLB players tend to be tall). Ranking = 500.

1 * 500 = Nothing. Nothing real can be generated from multiplying two rankings together.

Rankings are ordinal numbers. You can say that 1 is higher|lower than 5. You can't say that it is 5 times better|worse. (In pro sports, being champ, #1, is INFINITELY better than #2.) You can't say it is 4 better|worse. You can't infer any precise degree at all.

So: don't multiply ordinal (ranking) numbers. Make a matrix, sure. It probably is useful to rely on your subjective evaluation of where an asset fits (this has an impact or value of "9", that's a "3"). Then make a matrix of impact vs. vulnerability or whatever, and remediate accordingly. But DON'T use bogus math to drive decisions. ("This 4 x 4 = 16 is greater than that 5*3 = 15")

Now, without making up fairy tales about infinitely skilled attackers and such, you can generate some actual data for security performance metrics. Richard Bejtlich (who though way smarter than me (and probably you) is guilty of doing math on ordinal stuff in this post: http://taosecurity.blogspot.com/2003/10/dynamic-duo-discuss-digital-risk-ive.html) suggests the way to get real metrics on useful subjects is to do timed pen-testing and the like. Did it take longer for a skilled|unskilled team than last year? In other words, don't measure your team members' shoe sizes, look at the scoreboard! Here's the post: http://taosecurity.blogspot.com/2006/07/control-compliant-vs-field-assessed.html

This is hard, and expensive. But if you want useful metrics, it's what you do.

2006-09-25T11:50:00.000-07:00

Review of Designing Large Scale LANS, by Kevin Dooley

Good book! This is what the title implies: a book about designing large networks. It's not primarily an implementation book. It treats its subject rigorously, but without tons of detail at the end points. For example, you won't find cat5e pinouts discussed. You will see a redundant, heirarchical network design. I like a book with real math as , and the author actually provides some for aggregate Meant Time Between Failure (MTBF) calculations. Stats and probability! Cool! He gives less rigorous but useful rules of thumb for capacity planning.

Lot's of advice reflecting his extensive real-world experience. Like the importance of physically redundant trunk links (rather than just two circuits in the same fiber bundle|conduit). My impression was that stuff never failed unless a backhoe severed it, but I was...incorrect. Thanks! I will be working on a plan to get redundant links in place.

I had an intuitive sense that there is a trade-off between redundancy and complexity. Reliability is the goal, and you can add features (primarily redundant circuits and components) to a point where the complexity reduces reliability. Dooley gives a fairly clear impression of where the trade off is profitable.

The VLAN treatment is extensive. Again, I knew that trunking all VLANS on the campus net across all trunks was wasteful; he quantifies it.

Overall, the book stands up well after 4 years. He doesn't spend much more than a sentence or two on wildly obsolete media like 10Base2 (coax). There's the occasional PanAm moment (the shuttle taken to the space station in the movie "2001" is operated by PanAm) like when he refers to Compaq as a manufacturer of network interface cards. I still see issues with 10BaseT and probably you do too, so I don't begrudge him any space on the topic. He was forward thinking enough to mention gigabit ethernet. He refers to Cat6 cable as a future standard. He cautions against using intermediate patch panels, which I was given to understand are o.k. One major building on our campus uses them, at the behest of the wiring designer. Oops. I haven't noticed any problems, but now I know to look.

Wireless is the area where change has been fastest, I think. Probably something to do with inexpensive, commodity hardware (with broken initial specs) leading to faster refresh rates. He mentions (back in 2002, I remind you) the utter brokenness of the WEP encryption standard. But if wireless in detail is your thing, this is not your book.

There isn't much on different types of fiber optic cable. (not in book - this is my own accretion of data) What I know of is: single-mode has 9nm cores, and goes from 10km to 80km depending on the fiber transceivers. Multi-mode is in 50nm (newer(?) better distance|speeds) and 62.5nm (more common) cores. If you reach this page trying to see what the difference is etc., you can actually sub the multimode cable pretty freely. You will lose signal going from 62.5 to 50nm, but the optical power budget may support a connection even with the loss. Every splice and connection costs signal power. Every meter of distance costs signal power. Takeaway
is that SX transceivers (for multimode) don't care which you use, so you might as well install 50nm fiber. Single-mode transceiver vendors HP, Cisco, and Transition Networks use different names to designate stuff for the 10 km vs. 80 km stuff. For Transition, you have to look at the specs for particular units. They make a variety and call them all LX.

Vendor	10km	80km
HP	LX	LH
Cisco	LX/LH	ZX
TransitionNetworks-doesn't follow pattern	LX	LX

The IP routing/subnetting stuff is good.

QoS treatment is good: he shows why you can't just throw bandwidth at a problem to give good video|voice. Variable latency (called "jitter") makes it hard for voice|video apps to buffer, leading to pops and crackle that drive users up a tree. Of the three approaches, he recommends only Guaranteed Delivery will suffice.

Multicast treatment is good. I have never had a handle on that stuff. Now I do.

Some good operational details - in the network monitoring section, he urges us to monitor even quiet backup links. If the backup failed and nobody noticed, they will when the primary dies.

In sum, this book is worth the time to read it. It's a little old, but the stuff that is essential to its topic has not changed. Heck, the age just means you can get it dirt cheap. Check ebay or amazon used.

2006-09-25T11:47:00.000-07:00

Syslog-ng is the platform. Central log server

http://www.loganalysis.org/

Tina Bird runs a loganalysis list

Swatch sec.pl - simple event correlator lets you watch for and act on combinations of events

splunk - a search tool, the google approach to logs, rather than trimming and wading through

logwatch

logsentry

2006-09-25T11:43:00.000-07:00

AFS notes -

At a geek meeting (Seattle Area System Administrator's Guild, formerly Seattle SAGE)
randome recommendation to use heimdahl, not MIT)

2006-09-20T16:04:00.000-07:00

I went to a meeting of the local Infraguard chapter (http://www.infragard.net/) today.

Couple of interesting things: a presentation by one of the agents that worked on the zotob case. That case resulted in the arrest and conviction of a Morrocan citizen (and the ongoing prosecution of a Turkish citizen). It didn't hurt the investigation that the bots phoned home to a server in a domain named for one of the suspects. (Note to self: don't set up a botnet that uses irc.jimmythegeek.com for command and control. Other note to self: don't use the googlemaps link to my house for a domain name for botnet C&C, either. Other, other note to self: don't set up a botnet at all)

A Cisco guy gave a presentation on "Self-defending" networks, with the usual credibility- augmenting bashing of his own company's marketing department. Overall, I'd say there's a case to be made for multiple layers/levels of defense, all coordinated. The guy cited a competitor's approach (ISS?) that's "all about the math". No layer has to be perfect, if in the aggregate the layers reduce successful exploitation chances to near zero. There was a little magic security spray (http://www.ranum.com/security/computer_security/marketing/index.html) but I suspect the claimed 1,500 programmers/researchers are able to gin up some useful behavioral characteristics to alert on. It would take time I don't have to evaluate whether it actually worked well.

Besides, it's unafforable and annoyingly a-la-carte. Want an IPS? Sure! Just send massive ducats. Want reports out of it? That's extra. Want stats from the router or switch? That's extra. Ick. I don't want to spend another minute of my life managing licenses for tools to manage my real work.