For example, I just spec’d a ~$70K (retail) dual radio solution BelAir for a condo in FL. If we turn this opportunity to a MSO, then they get, say, everything for ~$50K with whatever monster MSO discounts MSOs typically get. There are about 100 condo units, so roughly….

$20/unit * 100 units * 12 mo = $24K/year the MSO earns.

So, in about two years, the MSO (in theory) recoups the cost of their CAPEX.

In cheaper condo environments (<$300K), particularly any sort of condo with only seasonal visitors, an open-mesh or Meraki sort of solution might make sense, as that way owners do not have to worry about monthly commitments. These units paying, say, $5/week or $20/mo on a per transaction basis pay off the infrastructure in a matter of months, with almost zero OPEX  (but with the added cost of having Meraki take their 20% for their automagical billing solution).

Open-mesh looks very promising, IMO. While I haven’t seen any billing solutions that integrate with it yet, I hear that several 3rd party solutions are on the near horizon. Rumor has it that both Meraki and open-mesh both have insanely easy-to-use monitoring solutions for their product. If that’s the case, I would love to see one in action, as this is something that is insanely expensive with other vendors.

On a slightly different note, I’m getting good reports from various associates and colleagues about various CPE gear, particularly the Mediaflex and Metroflex. Someone I know just mass rolled these out in an apartment in the Phoenix area, and the tenants he says seem quite happy.  As far as Meraki goes in this department,  a quick google search on “meraki openwrt” and “meraki netbsd” shows some interesting projects being kicked around (which I saved on my del.icio.us profile, for those interested)

After you install the plug-in, simply right click the folder with the dupes, and select “remove duplicate messages”. Le voila!

Any pointers in the right direction would be greatly appreciated!

On the third day (over 24 hours after the incident), I made a last ditch effort of just disabling the IMAP from within Gmail.  Sure enough, that fixed it about an hour or so later when I checked it.

I got a similar Gmail lockout a few months ago, that time for what I suspected was my Greasemonkey tomfoolery.  In that case, however, I was back in my account in a matter of hours, not days.  :/

Outside each hotel room, I would easily get ~12 Mbps, yet as soon as I came in the room (right by the window), my speed might drop to 9 Mbps, and on the first bed, I might get only around 6 Mbps on the first bed. The second bed might get 4 or 5 Mbps, and the sink area would drop to less than 1 Mbps. Bathrooms were the worst — around 300 Kbps or so. Interestingly, I almost never lost a continuous ping the whole time.

Today we easily light up the faces of several hotels with just one cheapo BA100 unit with the default internal antenna with a 9 degree tilt. Our second hotel today was a textbook case. One unit on each side of the 9 x 14 room area gave us solid coverage, and a unit on top with LOS to each unit provided the egress and backhaul to the downstairs units. As an added bonus, we found a perfect little water resistant cranny for the unit to permanently live on top of the roof!

In theory, it should have all been very simple.  All I did was upgraded from v23 to v24, but for whatever reason shortly after the upgrade, none of my wireless devices could use DNS properly. I downgraded back down to v23 and everything seemed to be working fine for a while, but then about 2 hours later I got locked out of my router (for no particular reason).  I did a hard reset, upgraded to v24, and everything has worked quite smoothly since.

One of the main reasons I upgraded was to support virtual SSIDs, so now I can have multiple SSIDs, one broadcasted SSID with WPA for me, and another one not broadcasted with WEP for guests.  I believe one can choose *not* to bridge that other SSID to the other networks current network, but I have not tried that feature yet.

One of the cooler features of dd-wrt is QoS, and for whatever reason, QoS on v24 seems to be a lot more functional than previous versions.  Using v24’s QoS, I labeled all torrent traffic as BULK, then I classified DNS, IMAP, SMTP, POP, and HTTP traffic as EXPRESS. For good measure, I labeled ssh traffic and my iPhone’s MAC address as EXEMPT, so now checking email during torrent-ing takes only a fraction as long.

As a proof of concept, I’m torrenting like hell right now, and my somafm.com stream hasn’t even missed a beat — something unheard of until today!

Insidder helps identify sketchy coverage (e.g. areas below -80 dbm RSSI). Its real time RSSI graph of the various SSIDs helps wrangle the 802.11 channel craziness by finding which channels are being tied up on neighboring SSIDs. Sometimes, if I have adequate coverage in an area and a laptop often jumps on a neighbor’s stronger SSID too soon, I might sometimes drill down into their driver and tweak the roaming aggressiveness. And for when you never want to roam to a neighboring AP using your SSID, I have begun started hardcoding MAC addresses to the actual SSID, which you can do in Linux as well as Intel PROset/Wireless program for Windows (haven’t done it in OS X yet).

I was talking with one of the other SEs in my company about the current state of wireless resembles the early days of ethernet (at least, based on what others have told me). Everything is a sort of voodoo, and finding good, simple to read resources / tutorials is not yet easy. In order to figure out the underlying protocol, you often have to sort through gobbledygook or hype, rather than going to a few key easy-to-read sources. As wifi gains momentum (particularly over wimax), this will change, I think. But until then, actually understanding the alphabet soup that makes a wireless protocol and its relation to applications (e.g. how collision avoidance affects VoIP) might be a little murky.

(1) Give all taco trucks a GPS unit, aggregate all the live data on some web backend, and then display it in some sort of a google maps mashup that’s phone-friendly (e.g. SMS your address, and it SMS’s you back driving directions from your address). Show me a single late night hispanohablante clubber who wouldn’t be all over esta mierda!

(NPR had a special a while back on a website that mapped out various taco eateries, but I’m really like to see something more real time that’s interactive with people with cell phones who want late night eating options.)

(2) Take all of the major datacenters in the United States (e.g. in SoCal, you’d take Hosting.com, One Wilshire, Savvis, etc) and then make, say, a wiki entry around each one that tells you about local eating and parking options. So, at One Wilshire, you might put The Pantry and then note that it costs $1 to park and that you may have to compete with people coming from The Derby. You might also note caffeine and parking options in the area, as well as primo parking spots for when you (a) don’t want to spend very much money, or (b) need to unload a lot of gear but don’t want to pay the $30 or whatever to park in the underground areas.

(Perhaps one of the first places you start with is some of the major carrier hotels that companies like CRG West own)

In one sense, I can understand how IPtables is a godsend, particularly in environments where network admins are slow to make necessary firewall changes or are (understandably) reticent about giving others access to networking equipment. On the other hand, ad hoc configuration kludges everywhere can get insanely unwieldy, and should the sysadmin leave unexpectedly (which I see all the time, particularly in high pressure data center environments), the next sysadmin who takes his place (and more importantly, the company!) is stuck with some major firewall craziness to sort out.

To deal address both of these problems, here are some solutions I have been testing in hosting environments:

• KISS My Firewall: a free iptables script designed for a typical web server (stateful packet inspection, connection tracking, some preventative measures for port scanning, DoS attacks, IP spoofing, etc). It is one simply one file, can be installed with stock installations of Ensim WEBppliance Basic & Pro, Plesk, and Webmin, and automatically leaves open FTP, SSH, SMTP, DNS, HTTP, POP3, IMAP, HTTPS, MySQL, Secure IMAP, Secure POP3, Ensim WEBppliance Basic/Pro, Webmin, and Plesk. Open ports on the OUTPUT chain include: FTP, SSH, SMTP, RDATE, WHOIS, DNS, HTTP, HTTPS, and OPENSRS. A few quick changes, and you can quickly close any of the defaulted open ports.
• APF: Also IPtables-based and with an intelligent modular architecture and detailed usage information (made available with the apf command). (A great HOWTO here)
• BFD: a shell script that parses application logs authentication failures. (Another great tutorial)

My friend Jeff uses these successfully more in his hosting business with great success, and I’m curious as to what other similar tools admins use to standardize their LAMP boxes.

In some way, shape, or form, I’ve been following wireless for a while now. I got my amateur radio license and try to keep up with various spectrum legislation issues, particularly on where I feel our public commons are being threatened. For a while now, I’ve been wanting to shift into the wireless space, and the planets and stars seemed to be in the right order for this decision. In our team, there are approximately 4 “wired” (routing/switching/IP) and 2 “wireless” (RF/cellular), with varying degrees of overlap between all of us. Lento pero seguro, we’re each learning the other side of the fence…

On my first week, I had to revisit a few issues I rarely have to deal with in any great detail.

• RSTP (802.1w): these mesh boxes are, essentially, layer 2 devices with antennas. It’s not uncommon to see hundreds of these things, so spanning tree issues can be quite a problem. Each box has a tiny version of Linux, so from what I can tell, the potential to making switching decisions based on higher layers (7?) is there.
• MobileIP: how you keep a given IP address while roaming around your network.
• IGMP snooping: how our layer two doodads intercept IGMP messages update their MAC tables.
• L2TP: this helps “flatten” different networks for seemless mobility
• 802.11e (QoS): I expected this to be similar to 802.1P, but from what I can tell, it’s not a tag like it is in the wired world.
• 802.11r (roaming): a lot of 11e got rolled into 11r, and I’m still figuring out what goes into making a Wi-Fi <–> GSM handover possible.
• 802.11 in general: I’m quickly learning what’s part of the overall standard, and what’s determined by specific standards or vendor interpretation.

A good example of some of this coming all together is seeing how the sausage is made when it comes to seemless roaming. If you think of it in wired terms, major things go awry when one unplugs a client from one ethernet switch and plugs them in another one: MAC addresses are now associated with different switches, traffic that was en route now has try again, etc. Roaming in wireless is perhaps no different, and when something moves from one AP (access point) to another, traffic en route is disrupted similarly. I searched the 802.11 specification, but cannot seem to find anything that dictates how the client should associate and disassociate from each AP; each client card (firmware, software, etc) seems to dictate everything. One great analogy I’ve heard is that bugs (the “clients”) stick to light bulbs (the “AP”) until they die (i.e. signal fades away), then they begin to look for another AP to associate with.

Up to this point, I’ve pretty much taken for granted what vendors tell me about their IEEE implementation, either in their documentation or in their propaganda disguised as certification. In one week, I’m noted several interesting situations where vendors (not us, of course!) horribly simplify something (”seemless roaming”), is perhaps intentionally misleading (e.g. “full duplex wifi”), or implemented so poorly that it’s intended benefits are outweighed by some other crapiness (e.g. multiple radios in a box interfering with each other).

Today I happened to looked over my boss’ shoulder while he was configuring one of our wifi switch routers and noticed that he was using TFTPD32.

I downloaded it and was jazzed about seeing the DHCP, TFTP, SNTP and Syslog functionality that it has built in. Simply put, that means that when you reset a unit and it, say, defaults to DHCP mode, you can simply use this little exe program to quickly assign it an address, see the address that’s being assigned, and then rock and roll! Before this, I was doing all sorts of other craziness, such as nmap’ing the DHCP range of a network and then looking through the MAC addresses to find my unit.

You gotta be careful, though. Rumor has it that your computer supports RSTP and if you were, say, on a customer’s WiFi network *and* wired via ethernet some units (as our muni wifi boxes), you could run into problems and create some sort of bridging loop or something. (I have yet to confirm this and am still getting up to speed on RSTP issues in production).

TFTPD32 allows you to tweek tsize, blocksize and timeout, which (in theory) allow you to maximize performance when transferring large amounts of data. Haven’t tried this, but would be very curious if others here had…

You served me well, protecting me and often keeping me company during some cold nights in the data center. Remember that time I blasted you with the SmartBits because I doubted you? It seems like almost yesterday.

The only rocky point in our relationship was one armed routing, but looking back, it wasn’t wasn’t that bad, and you later worked out the kinks in 7.0. I’m sorry I later switched to CheckPoint. I hope you understand. I tried to still include you as much as possible, but CheckPoint’s friend, Nokia, just didn’t think you were cool enough to hang. Nokia brought up that one time when you crapped out when I pounded you with the SmartBits, and I didn’t stick up for you like I should have. I wish I was a bigger man then, but I wasn’t.

I like to think that you’re in a better place now.

Google Calendar integration with the iPhone is definitely a mixed bag: type in (something like) “lunch with friend noon today” in the “quick add” field, and it will automatically enter in the appropriate time; try to edit or delete that same entry within Google’s mobile pages, and it’s a no go.

It will be interesting to see how 4G drives IPv6 adoption.

(On OS X, I’ve been using Burn and have been quite impressed.)

Those wanting an open source alternative to Adobe and FoxIt Reader should check out Sumatra PDF. For only 500K, it packs quite a punch, particularly for lower end PCs that might need lighter tools.

(More tips like this can be found on the Adobe Gallery of Remedies.)

• change contact_groups from “admin” to “admins” (”admins” is probably already your default group in localhosts.cfg)
• define contact_groups in the host definition, rather than the hostgroup definition (based on the default config, this can be done by using “use linux-server” in your host definitions)
• edit address line (e.g. put in valid A record or IP address)

I threw a quick HOWTO on NagiosWiki, and eventually I hope to get to redoing that perl script to be compatible with 2.x/3.x cfg files.

• wget -r -l2 -t1 -nd -N -A.zip -erobots=off http://sdsickboy.com
• unzip ‘*.zip’

Now play the unzip’d .flac and .mp3 files in your music player.

• likes punk music,
• has a JD/esq (Escalante from Loyola; Orlando from UCLA),
• has a hispanic sounding last name,

and perhaps most importantly,

• likes to cautiously give half baked legal advice!

While I don’t know jack about podcasting, I urged him to consider creating a domain name and general theme, and once those were set, see what follows.

Mike is the type of dude who lives life in order to narrate it. I can only imagine what could happen if you gave him a soapbox, cheezy intro music, and some soft pitch legal questions on what the law says about building nonprofits.

• …can use F/OSS tools to build out crazy monitoring infrastructures (think tens of thousands of service checks).
• …is wanting to work in a fast-paced integrator environment (with sales people who find monster opportunities).
• …can communicate well (i.e. be able to respond to an RFP or create a solid Scope of Work).
• …is “plugged in” the F/OSS community (e.g. interacting with the people who wrote the source code for the tools which you’ll be using).
• …is adept at scripting and ok at programming (enough to read through and possibly modify other people’s perl and python scripts).
• …learn whatever technical skill they have to in order to “make it happen”.
• …can think creatively on how to solve “impossible” projects.
• …is able to interact comfortable with “c-level” executives (e.g. CTO, CIO, etc)
• …is willing to work in Newport Beach, CA.
• …is willing to travel once in a while.
• …can conduct [him/her]self like a professional.

Email me (Roger @ Hack My Idea . com) , we’ll chat on the phone, and I’ll give you the scoop and some (hopefully!) some helpful tidbits. After we talk, I’ll shoot your contact information and resume to the main decision makers.

I am NOT a headhunter, nor do I get any sort of financial compensation for this referral. I’m just looking to help an integrator I used to work for find a solid candidate. If nothing else and you’re not the best match, shoot me an email anyway, and maybe we can expand our list of professional contacts. If there is one thing in my career that I’ve learned, it’s that good people know other good people!

Next on my radar is a series of Cisco ASA HOWTOs (similar to the PIX series I wrote for Techsoup a few years ago). Compumentor just drop shipped me some equipment, so that should be cool…if for no other reason my professional contacts tend to be CheckPoint resellers!

For a while now, I’ve thought it would be cool to make an equivalent using sites like LightReading.com, Clay Shirky, etc.

I was quite pleased to stumble across this OPML file of all of NPR’s podcasts. (Here is another one on OPML.org that lists them in a more rational format.)

I went to CrazedList and put in one search term that spanned across all the relevant CraigsList sites (mostly sites in Southern CA, but some were nationally as well). I then put the CrazedList -generated OPML file in FeedMagick2’s “Inspect Pipeline: OPML Reading List Blender” and used the resultant RSS feed as my input value for Y! Pipe’s “Fetch Feed”. Using the “union” and “unique” operators, I could aggregate “Fetch Feed” inputs and parse out dupes.

(I probably spent 4 or 5 hours looking roll my own ghettofabulous ‘OPML to RSS’ parsing solution before throwing in the towel and just googling for an online tool to automagically do it for me. Please post any cool solutions you have in this area! Programming is not my primary skill, so go easy on me…)

Those wanting a “play by play” on how it works, can follow these steps.

1. Go to www.crazedlist.org and enter in your search string and terms (I’ll use “linux” as the search term, check “resumes”, and select “all” for regions)

2. Once we click the orange “get RSS feeds” button, we’ll get something like this.

3. Up near the top, you’ll see something like this

To get the OPML file right-click on this link to Save Link As…. when doing this you’ll be prompted to save a file called index.cgi or index.xml or something, you can change that and name it something like “mysearch.opml” then import that into your favorite RSS reader that supports importing feed list.

4. Open up FeedMagick2’s “OPML Reading List Blender” and enter in the path to the aforementioned OPML file. Convert that to RSS. (This may take a while). Note that path to that RSS feed. It will be your input for Y! Pipes.

5. Log in Y! Pipes, drag in the Fetch Feed module, and copy in that RSS feed. Link that to whatever modules you want, and publish your feed.

(Step 5 may seem unnecessary, but it shortens the path of the RSS feed considerably, making it more accessible to RSS readers which might not take such a long string)

While talking to my friend Chris about some ideas I had for Debian packages, he pointed me to a cool project called CheckInstall. This little tool looks awesome, especially considering I’m not that familiar with Debian’s way of doing things. Once your ‘make install’ is done, CheckInstall will create a Slackware, RPM or Debian compatible package and install it with Slackware’s installpkg, “rpm -i” or Debian’s “dpkg -i” as appropriate. According to the website, this script leaves copy of the installed package in the source directory so that you can install it wherever you want.

Hmmm…cool or dangerous? I’ll have to look at how exactly it operates and see if it really does everything necessary to make a “proper” .deb package.

This tutorial is a good quick overview on what it is and how it makes you more productive with multiple ssh sessions, particularly in environments where you need to collaborate with others, document your session, or make sure that a disconnection will not stop whatever process you’re running.

One frustration I have is not being able to go to the beginning of a line (control-a) while in screen. Any suggestions there would be greatly appreciated!

The Nagios class turned out to be quite a cool experience. I had about seven students who came from SAIC (1), Horizon Technology, (1), D-Link (4), and RK (1). LAMP skills between students varied widely, and I’m hoping that everyone left the class with something significant. Some of them have joined our LinkedIn group and have been networking with others in our group.

For what it’s worth, here are four more integration projects that I started this week. I will iron out HOWTOs on NagiosWiki once I get some time. (Those wanting specifics beyond what I have listed are welcome to email me, but please realize that I may not have all the kinks worked out yet)

1. ticketing and Nagios: once Nagios detects a down host or service, email OSTR (open source ticket request system) and autogenerate a troubleticket. When that host or service comes back online, the email on status change closes out the ticket. Once I iron out a few wrinkles, I’ll integrate this into RT (which I think is “better” in some ways).

2. Monitoring for ssh key corruption: When ssh keys get corrupted and need to be regenerated, check_ssh will not detect the login error (to my knowledge, at least). I’m not a Perl programmer, but I’m hoping that something in Net::Telnet (which can be told to use SSH for the underlying transport) or Net::SSH can help me prove that a login is failing on a few thousand routers. (Still googling for what others have done in this department. Any ideas here would be greatly appreciated!)

3. service reliability checks using NagiosPluginsNT: If you’d like to run a check *from* some weird nook and cranny in your network and do not want to deploy a Linux box with NSCA so you can relay passive checks, consider doing the following:

a. installing NSclient++

b. dropping the NagiosPluginsNT plugins in your c:\path\to\nsclient++\scripts directory

c. modding your c:\path\to\nsclient++\nsc.ini file to include

check_http_google=C:\Program Files\nsclient++\scripts\check_http.exe -H www.google.com

(Of course, test this from your Nagios server - “check_nrpe -H windows-box -c check_http_google“)

4. check_disk on /proc/mounts: started adding the following NRPE handler in the nrpe.cfg of various Linux servers with weird disk partitioning.

check command[check_disks_proc_mounts]=/usr/lib/nagios/plugins/check_disk -w 15% -c 10% $(for x in $(cat /proc/mounts |awk ‘{print $2}’)\; do echo -n ” -p $x “\; done)

(I had horrible problems with this command yesterday, as vim commented out certain sections, I wasted time trying to escape those characters. Muchos grassyass to my buddy Ed for helping me debug this one!)

Traditionally, one would just run “fdisk -l” or “df -h” and then write a separate NRPE handler for each command. In environments with crazy partitioning (or, better yet, NO partitioning!) or crazyass LUNs volumes, you gotta just send one command that checks the collective health of everything and then reports back if one of those volumes has exceeded its critical or warning level. If that server is important enough to check a particular volume or media for a specific parameter, then consider hard coding a specific NRPE handler for that server.

5. DNX + Nagios: This project offloads active checks to worker boxes, saving you (theoretically) lots and lots of time changing active checks to passive ones via NRPE and NSCA. I just untar’d the project and have been reading over the documentation. It looks easy enough to integrate, but I’ll know more when I put the rubber to the road.

The bottom line to these projects: automate, automate, automate! If you have to do it once, then do it manually. If you have to do it twice, do it manually *and* document. If you have to do it manually a hundred times? Nigga please….automate, yo!

I found fping in these situations to be the “magic bullet” to help me start narrowing down DNS problems.

e.g. something like: fping -d -g 10.0.0.0/24

-g would let me feed in a weird subnet (or range of IPs), and -d would turn around and ping the host name that the ICMP reply gave me, ultimately giving me the A record someone needed to BIND to help with the transition.

From there, I could use the standard unix tools (grep, sed, etc) to sort whatever output fping crapped out.

Anyone have a “better” way of doing this?

(Just who do they think that their market is with that description?)

(1) grab the pre-compiled binary, unzip, and then untar it in the /opt directory, which is where many admins like to store large packages.

cd /opt
wget http://www.domain.com/path/to/nrpe-nsca-plugins.tgz
gunzip /opt/nrpe-nsca-plugins.tgz
tar xvf /opt/nrpe-nsca-plugins.tar

(those who’d like to simply look at the contents of the tar can type tar tvf)

(2) Now grab and configure the /etc/ files. Some admins (like myself) like all the /etc/ config files to be in one location.

cp /opt/nagios/etc/* /etc/

OR log on another similar box and type in the following to push it over to this box

scp nrpe.cfg root@otherbox:/etc

OR grab nrpe.cfg from another box and copy to your /etc directory

scp root@otherbox:/etc/nrpe.cfg /etc/

(3) modify the NRPE handlers in nrpe.cfg to include the correct path to the binaries you gunzip’d and untar’d in /opt

command[check_users]=/opt/nagios/libexec/check_users -w 5 -c 10

(4) now, add Nagios’ NRPE to the /etc/rc.local for each reboot

nohup /usr/bin/nrpe -c /etc/nrpe.cfg -d

(5) Now, start the NRPE service and associate it with the /etc/nrpe.cfg file you cp’d over

/usr/bin/nrpe -c /etc/nrpe.cfg -d

(6) Now, double check everything to make sure it’s all working. That includes: (a) checking the running processes on your AIX server, (b) checking the rc.local file on your AIX server, and also (c) making sure that your Nagios server can access the AIX server using its check_nrpe plugin. If there is a problem, try tail-ing the syslog for clues (tail /var/adm/syslog)

less /etc/rc.local
nohup /usr/bin/nrpe -c /etc/nrpe.cfg -d > /dev/null 2>&1 #Start NRPE for Nagios

AND

ps -ef | grep nrpe
root 458886 589858 0 01:11:51 pts/0 0:00 grep nrpe
nagios 536778 1 0 01:11:13 - 0:00 /usr/bin/nrpe -c /etc/nrpe.cfg -d

From Nagios (this was on my CentOS 4.x box)

cd /usr/lib/nagios/plugins/
./check_nrpe -H AIX_Server -c check_whatever
(RETURN OK)

Thanks to Everyone Who Pitched in to Keep Our Podcast Free

A few weeks back, we asked you to help our home station pay the substantial bill for offering our show as a free podcast and online stream. A huge thank-you to the thousands and thousands of people who contributed: in a few, short weeks we raised enough to cover all the bandwidth we used in 2007. All of us at This American Life and Chicago Public Radio thank you, again, for your generosity!

Of course, it’s always been free to the technorati who have know the secret URI to download whatever episode they’ve wanted. To find it, fire up Wireshark, start it on the interface that is connected to the Internet, and you will see a GET command in the INFO field similar to the following:

HTTP GET /jomamashouse/ismymamashouse/(EpisodeNumber).mp3 HTTP/1.1

So, in layman’s terms, you can simply put the following URI in your browser window to download and listen to past episodes!

http://audio.thisamericanlife.org/jomamashouse/ismymamashouse/(EpisodeNumber).mp3

(Or, if you have Linux or OS X, go to the command line and put a wget before the URI. A win32 version of wget can be found here)

Several months ago, my friend Chris at LSU’s SRCC recommended me lftp. I’ve used it quite extensively on Linux servers. It’s a wonderful quick and dirty cli tool, particularly when quickly scripting out FTP PUTs or GETs on Windows servers is involved. (Here is an excellent tutorial for newbies)

Today I found a Windows binary of LFTP on the Doom9’s forum. I scanned it with up-to-date AVG, ClamAV, and Symantec signatures, and it all seems to be kosher.

Those wanting to see how it works before they download it might appreciate the following lftp –help output.

lftp –help

Usage: lftp [OPTS]

`lftp’ is the first command executed by lftp after rc files
-f execute commands from the file and exit
-c execute the commands and exit
–help print this help and exit
–version print lftp version and exit

Other options are the same as in `open’ command
-e execute the command just after selecting
-u [,] use the user/password for authentication

-p use the port for connection
host name, URL or bookmark name

Personally, I do it all with the following type of command:

lftp ftps://user:password@ftp.yourserver.com:990

(From there, you can quickly navigate through the folders like you’d expect to normally in in FTP.)

Those paranoid about installing anything on a stable box should probably closely examine the hashes and contents of the install.bat file.

lftp-install.zip ea72152c91b52ac7ae5f962764eef4eb

install.bat

copy *.* %windir%\system
regsvr32 %windir%\system\cygwin1.dll
regsvr32 %windir%\system\cygncurses5.dll

lftp.exe 8be4378428af8a18c9d1cc58abac1241 (md5)
cygncurses5.dll 2044dbb9e65b51d5345be7d62b27477e (md5)
cygwin1.dll 596938bda60e655b82a68adb315e3bc6 (md5)

To see what he was made of, I simply asked him if he ever had to observe the 5-4-3 rule. He had never heard of it. And when I told him what it was, he then told me that was no longer needed. And when I told him that to my limited understanding it was needed “back in the day”, he changed the subject to vampire taps! Hmmm…go figure. (So maybe it wasn’t just me when we seemed to have a little disconnect between the difference between a broadcasting and a collision domain.)

Hands down, my favorite posers thus far are the “forensic experts” I’ve screened. I don’t know beans about forensics, but I have a clue when it comes to certain key concepts, and if pushed, I could fight my way out of a paper bag on any of those topics, as well. One “expert” said that he had all of these forensic skills, and when I finally asked about how exactly he acquired the data before he analyzed it (one of the few forensic topics on which I’m somewhat knowledgeable), I got an answer on how he didn’t do it. When I pushed him on the nitty gritty on how he “forensically searched” the contents of people’s copmuters, I found that he had done little more than install EnCase on his computer and conducted extremely rudimentary searches on some EnCase images.

Being a sort of technology generalist, I can certainly understand that no one person can understand everything about a given subject. It’s often easy to get caught not knowing very basic things about something you’re supposedly an “expert” in. But when it’s obvious that you’re batting WAY out of your league, c’mon…

Here is a little command that helps when you must mass move tons of files from one NT box to another. These funky switches are important, as they help grab all the subdirectories, hidden junk, and meta-crap.

xxcopy x:\folder1 y:\folder2 /s /h /tca /tcc /tcw

If I’m feeling bitchy about hitting Y too many times, I like add the /yy command. At the end you’ll have a nice little report that tells you what hiccuped.

Often times I’ll find some cool little hack tool to run against a bunch of files, and that little hack tool does have good folder recursion features.  A quick kludge in these cases is to just have xxcopy “flatten” all the directories in the destination folder and then let my hack tool go to town!

1. Have all sorts of weird aliases, such as info@domain.com, etc.
2. Use Exchange 5.5/2000/2003 (none on 2007 yet)
3. Use different clients to access their mail (making it difficult to quickly solve the problem via one of my favorite Outlook spam plugins, Cloudmark)
4. Have a Crackberry and/or iPhones (which beeps incessantly because of all the spam).
5. Refuse to immediately buy a real solution
6. Are almost solely responsible for the server queue with thousands of spam messages, which of course screw up Exchange’s Information Store jeopardizes the health of the mail server. (And me constantly running to the Exchange server and to type (aqadmcli and then delmsg flags=all is NOT a solution just because it “worked” last time)
7. Don’t want Linux in their organization and/or don’t have a spare server for me to throw Postfix on.

So, as long as the email sent is some sort of alias (and not their user name), you can often just kludge the following solution:

1. Create an Exchange forwarder for that alias@domain.com address
2. Setup Gmail to relay back to the internal address of the Exchange server.

Ghettofabulous, I know. But sometimes that’s how you gotta roll!

Everyone is worried about losing email in this process, but honestly, the biggest risk in this is it working so well for these VIP types that they want you to now do that hundreds of times for all the other users!

File before: 3,469 KB
(File after: 1,473 KB)
—————————————————
crap no one needs:  1,996 KB

So, there’s this huge push to make sure (a) stuff says on NTFS drives (and doesn’t wonder to FAT32 drives), or (b) someone doesn’t boot to a live cd and bypass NTFS permisssions, but what happens within NTFS and Active Directory to make sense of the madness? I’ve done a lot with Active Directory, and every time The Big Cheese asks me who can access what, I cringe. Doing it within Windows can sometimes be quite a chore, and even when I use third party tools such as AccessEnum to make that process less painful, it’s still sometimes a big chore. I can remember a couple of years ago having to nitpick NTFS permissions for a biomedical client of mine who was paranoid about unauthorized document changes on engineering specs and having to set up groups / users and then use all sorts of auditing tools to make sure that I (or another admin) didn’t give too many permissions to various users.

One solution to this scenario might be with ‘data governance’ tools like Varonis. Using their solution, you can do lots of things that’s a HUGE pain in the ass with native Active Directory (AD) tools, such as creating users and permissions for specific time periods, creating changes in a sort of AD sandbox, finding dependencies on AD objects, etc. I met with the channels rep from Varonis the other day and was quite impressed with their demo. Using their product, someone in a business unit could, say, add new permissions for contractors on special projects, and once x out of y people approved that addition, the contractor would have AD permissions for Z amount of time. Or, say that someone needed to clean up AD (something that’s always a pain in big organizations), they could turn on Varonis’ reporting (which is way less resource intensive than Windows’ native file auditing, which sort of assumes you know files or users are suspect). Lots of cool stuff there….

I was impressed with how many bases Varonis covers. From a C-level perspective, it puts business units in the driver’s seat. In too many companies, IT departments (for good or bad reasons) cripple business functionality. From a tech “trenches” perspective, I’ve now got a tool that helps me do easily what was previously quite arduous and take care of duties that would otherwise have fallen in the cracks.

Pricing on Varonis is a sort of combination between these probe clients installed on file servers, as well as AD users. As you might guess, it is insanely expensive for the the average company, leaving it up to places that are willing to pay huge premiums to ensure that they’re SOX compliant.

(Soon I should have a demo running on my VMware server, and perhaps can have a better review at that time.)

While I haven’t yet used Ruby-based AutomateIT in production, I wonder if it (or something like it) will be soon be key to easily automating tools like Procmon and documenting their effects on Windows systems (e.g. DLL calls, registry hooks, and dependent processes). Server automation tools (e.g. CFEngine, Puppet, etc) are key for many business reasons — reducing risk / errors / downtime, simplifying updates / migrations / recovery, codifying knowledge into repeatable “recipes” (sudo make me a sandwich!) — but perhaps soon we might see them integrated into incident response toolkits?

I don’t see a demo on their site, but I do see copies on YouTorrent and Torrentz. (Not sure if they’re authorized copies, though)

Because of a screwy cd drive on the laptop, I couldn’t just boot into the Sid / Knoppix-based Helix CD and use LinEn to image, but rather had to remove the hard drive, break out my newly purchased UltraKit write blocker kit, and image it that way (which, while more forensically sound, takes much longer to do).

Overall, the Ultrakit is fairly straight forward. In one side of the Tableau IDE write blocker, you plug in an IDE cable, which plugs into the IDE converter for your little IDE laptop hard drive. In the other end, you plug in your A-A USB cable, which plugs into your computer’s USB port. In the computer’s other USB port, you plug in your target USB drive (a 500 GB Seagate, in my case). Lights on the Tableau clearly tell you that everything is working ok (disk activity, connection to host, write blocking enabled, power, etc).

About an hour into imaging the laptop hard drive, I decided to kill the job and start the process on a laptop that was significantly faster. Bad idea — something I should have know better than do! Unplugging and rebooting Helix must have resulted in some sort of dirty dismount, because after booting on the new laptop and mounting the same Seagate 500 GB hard drive with ntfs-3g, I couldn’t seem to write properly to the drive without some sort of “input/output” error. Instead of using Sid’s ntfstools to “fix” the problem (which I’ve done in the past), I just used an XP Pro workstation to quickly format it again.

I then tried to use my Intel-based Macbook Pro for the imaging, but for whatever reason, Helix wouldn’t boot past the basic Grub menu. Instead of trying to figure out why, I just finished the job on my HP laptop.

For those new to the process, here are some steps I did to image the drive (most importantly, I guess, is NOT to mess with the connection once you start the image transfer!).

(1) Quickly format the destination USB hard drive to NTFS. Since I don’t completely trust the NTFS tools, I most always use Windows. (One recommended step before this is to zero out the hard drive in Helix by typing [something like] dcfldd if=/dev/zero of=/dev/sda bs=8k conv=noerror,sync, of if you want to securely wipe it, type something like wipe -kq /dev/sda)

format e: /fs:ntfs /q

(2) Once in Helix, check out which devices the operating sees (but may not have mounted).

fdisk -l

(3) Say it’s /dev/sda (I sure hope so, or you’ve zero’d out the wrong hard drive!), let’s make sure that it’s completely dismounted. If not, then we will need to do close or kill anything else in Helix which may be accessing that USB drive.

umount /dev/sda1

(4) Create a folder that we can later mount to, as well as a folder within it that we can put in the forensic images we get from LinEn.

mkdir /mnt/bigUSB
mkdir /mnt/bigUSB/forenicImages

(5) Now, let’s map that USB device in /dev to the aforementioned folders.

ntfs-3g /dev/sda /mnt/bigUSB

(6) Once you mount that folder, you are ready to acquire an image with LinEn. Start it and select the /dev devices which you are imaging (not the hard drive that will be written to). Now, when you enter your save path, enter:

/mnt/bigUSB/forensicImages

(7) Enter in all your forensic notes, and then sit for a while.

Le voila! On fairly mediocre laptop, it took about an hour for every 20GBs. Afterwards, you can use the hashing tool to make hashes of the attached hard drive to ensure that nothing was lost or altered in your images.

While I used NTFS (because I had some ~50 GB EDB file), it’s important to note that Guidance Software, the makers of Encase, suggest using FAT32 on the target drive (which can be done by typing something like mount -t vfat /dev/sdb1 /mnt/bigUSB). I suspect that this is a result of portability, as FAT32 is a sort of “lingua franca” when it comes to file systems. Also of interest to forensic types is the wide array of other tools out there for making Encase images, some of which claim to make Encase images faster than LinEn.

Update: One extra step you should perform when doing forensics is computing and writing down the hash of the entire partition (this, of course, assumes that sdb [2nd USB drive] is your ’suspect’ drive, not your target drive).

e.g.

md5sum /dev/sdb
sha1sum /dev/sdb

One of his tactics is to pick particular events (which, if taken into context, are probably what any reasonable person would have done) and then hammer away on people. By selectively picking something “stupid” and constantly revisiting that “stupid” mistake, he successfully beats others into submission so that they are less likely to resist in the future. He bolsters his legitimacy to do so by claiming that president of the company holds him personally accountable for everything under him.

In light of this mess, I couldn’t help but mass send this Frontline PBS episode to most everyone in the department. My favorite is the end (clips 4 and 5), where the correctional officers are put through the same exercises as the 3rd graders. In situations like this, the “inferior” group rarely (if ever) wins. If they argue, then they’re exhibiting the “inferior” traits that they supposedly have, and none of the others in the group will do anything but sit on the side lines and hope to eventually be a part of a winning team.

Perhaps the most bitter pill in all of this is that in the end, this yahoo isn’t doing anything that his employees aren’t letting him do.

I have no idea if the following video below is staged or not (why are the main people mic’d?), but it captures the build up of these tantrums well: someone is late to some sort of conference (which seems to always be held in a hotel), the main sales / biz dude gets really restless and starts making a scene (on little things like empty coffee pots), and then starts slinging things and barking at random people.

I’ve seen this multiple of times in the technology industry with various sales people, and for whatever reason, these yahoos still moving up the food chain. And when you talk to someone at a different company who worked with them, it’s always great to compare their tantrum stories!

Enjoy!