Tag Archives: System Admin

Using Windows Performance Toolkit to find System Issues in Vista/Win2k8/Win7

Windows 7 RC1 just came out. I am a TechNet subscriber, so I wanted to try it out. I have an old (2005) Dell desktop, 2.8 GHz, 2 GB ram, 160 GB drive box. 3.7 rating for Vista (because of the Graphics card mostly, would be 4.4 otherwise – not too bad, even for being kind of an old box). It has been sitting in the basement since I moved into my new place in October, doing nothing really. I use Mac full time at home, so it just sits.

A few times I have tried to get Windows Vista running smooth on it, Media Center, or just a file server,etc. Thing is, it was just flaking out. I knew it was a hardware issue, but figured it might be the CPU fan, or overheating, etc. Vista installed fine, but as I was using it, I would see just hang-ups, lockups. Not BSOD’s, but it would just hang, for 30 seconds, 1 minute, and then come back. WTF?

Nothing in the Reliability monitor, nothing I could see in event logs, etc. I rebooted, did Windows Memory test, nothing there. If you go into Computer Management, you will see Performance, then Data Collector Sets and Reports, Monitoring Tools. You can set it up to run a test on metrics of your system and it will give you a report

image

I did this, and everything was ok. BUT… Avg Disk Length Queue was > 2 – red flag. Disk issues. But I wanted to know more. So I started digging around, and there is a Windows Performance Toolkit you can download. Here is another good site going into detail about the WPT.

So I fire up cmd line (as admin! – start->run, cmd ctrl+shift+enter), and run

xperf -providers K

to see what providers are available for the Kernel flags. IOTrace looks like something I want, so I then run

xperf -on IOTrace

and let it run. I go and open/close things, play around, see if I can replicate the issue. Once I feel I have, I want to stop and analyze the trace. You need to stop it and output to a file using this command:

xperf -d iotrace.etl

Side note: Files are named ETL. Coming from a BI background, this makes my world explode, since it has nothing to do with Extract, Transform, and Load

Now that my trace is done, time to analyze:

xperfview iotrace.etl

And you get some awesome stats like this:
image

Although I didn’t save my stats from my tests that showed the bad IO, what I saw were just gaps in the graphs, glitches in The Matrix. Time missing. Something is really bad here. So I did the drive error checking in Vista:

image

And when that ran, after reboot, it got to 11% and croaked. Bad drive. So I went and bought a new 500 GB SATA drive and loaded it up, and I am running Windows 7 now. Pretty sweet.

After all this fun spelunking into Windows performance, it also got me thinking about things, like running these detailed traces on SQL Server boxes or other servers on intervals, and saving them somehow, reporting on the data. The IOTrace is just one of hundreds of traces, that you can then auto analyze. I know that there are perfmon tools but there are some added benefits to xperf that you can you utilize, and I am glad I learned more about it and put it to use, just another tool for the sysadmin tool belt.

HowTo: System Admin – Remotely Log Memory of Processes Greater Than X

If you ever wanted to remotely log memory usage (and CPU usage) of processes, you can use "tasklist"

The key is, "tasklist /s"  and then the remote computer name. That along with filters you can query and log process info remotely with no more than a cmd window, or batch file in a scheduled task.

For example, here is how you would log processes with memory over 25000 KB

tasklist /s remotecomputername /fi "memusage gt 25000" >> c:mem.txt

take a look at the tasklist help by typing "tasklist /?" at the cmd prompt. You can see how all the filter work. You can filter by PID, Image Name, etc. Pretty sweet. BTW it does work on your local machine too, just remove the /s computername

How To: Connect to SQL Server, VS TFS, etc using Windows Authentication when computer is not on Active Directory Domain (XP and Vista!)

Whew, long title, amazing results!

Problem: You have a laptop or computer and you are working remotely for a company. You VPN in. Your computer is not on their Active Directory (AD) domain. You try to connect to SQL Server using SSMS or Analysis Services using Excel, but it doesn’t work because it is using your user, not a domain user. How do you get around this?

Answer: Well, this is what I have found (tested on XP only) – start->run: \computernamec$ – then it prompts you to login. Use your AD username and password, so

domainusername and password, and check the box to save password.

Seems that XP will save that in your authentication list somewhere, and then you can use SSMS or Excel to connect to the SQL Server via Windows Authentication!

This trick also works for TFS Build Servers/Team Explorer (tested with VS2005 Team Explorer) ..

Now for the fun part – Vista. The tricks above don’t work on Vista, but you can still get it to work. Here is what you do…create some shortcuts…

C:WindowsSystem32runas.exe /netonly /user:domainusername “C:Program FilesMicrosoft OfficeOffice12excel.exe”

C:WindowsSystem32runas.exe /netonly /user:domainusername “C:Program FilesMicrosoft Visual Studio 8Common7IDEdevenv.exe”

C:WindowsSystem32runas.exe /netonly /user:domainusername “C:Program FilesMicrosoft SQL Server90ToolsBinnVSShellCommon7IDESqlWb.exe”

 

Replace “domainusername” with your info. So if your domain is mycompany then it would be mycompanysteve.novoselac for example.

What happens is that then when you run those apps from those shortcuts it will prompt you for your domain password, you put it in, and it runs the app in the context of your domain user. You can then change the icon for each of these pretty easy, just browse to the exe in the second part when clicking the change icon button on the shortcut properties (the shortcuts are actually links to runas.exe which is a generic icon)

In Vista, for instance, if you are testing SQL (SSMS), you might get this error:

Login failed for user ''. The user is not associated with a trusted SQL Server connection. (Microsoft SQL Server, Error: 18452)

The shortcuts above will get you around it in the situation where your computer is not not on the domain or you are not logged in as a domain user..

These tricks above are especially good if you need to connect to SSAS (Analysis Services) since it is only Windows Authentication. And also, the IT department doesn’t really need to have consultant machines on the domain, or VM’s set up, etc, instead they can use these workarounds

IT Disaster Recovery, what the I-35W Bridge Collapse shows us

Now, I am from Minnesota originally. I drove over that bridge 3 days before it collapsed. It sucks, its a bad thing for the state, for the people involved, and for everyone who passed or is injured. It is a very sad situation that no one should have to go through.

What does the bridge collapsing have to do with IT? Well. It is a disaster. And like IT disaster scenarios, it gets the same “Oh my god we need to fix this” after the fact treatment.

MN Gov. Pawlenty announced an immediate emergency round of inspections of all of the state’s bridges, starting with the three that have the same structure as the crumbled Minneapolis span. Other Gov’s are having their bridges inspected in their states. People are running around going crazy about inspecting bridges that 3 days ago they could care less about. – What gives?

Really? Lets do something after the fact. The bottom line is that these kind of action plans should have been set up beforehand. Just like in IT. Backups is a good example. No one says or does anything or wants to spend any money on backups. Then one day, the server crashes and everyone loses their files and email. I will bet money the next day there is a huge budget and people running around like idiots getting a backup plan in action.

Where were those people before hand? We know that stuff needs to be backed up. We know that bridges need to be inspected. WTF are we doing? If we know the possible problems, and we know how we can prevent them, then why do we let things slide. Where is the accountability?

The government needs to step up. People that are leaders/decision makers need to step up. And if something does go awry, they need to take responsibility for what happened. Wether it is a bridge that fell, or a server that crashed, or any other disaster scenario.

Real World IT: Backing Up Cisco Routers using .NET

Usually, in a company, there is a “development” department and a “IT” department, and usually the departments don’t really work together. Development is focused on delivering business value through coding applications for end users or B2B clients. IT is busy making sure the corporate network is humming along, and that other internal issues related to technology in general are taken care of.

In my experience, I like to jump the threshold between the two departments. I started out working Helpdesk (IT Dept) and coded in the time I had free, eventually starting/breaking into the Development side. But, my passion for internal IT functions didn’t slow. Some of the guys I worked with in the IT Department always wanted applications to do specific things for OUR network, things you can’t buy, or things that you can buy a generic application for way to much money and it won’t work exactly how you want it to. That is where developers and IT can actually work together and bridge that gap.

This post is about backing up configurations on Cisco routers using .NET (C#). Now, most developers programming away on business applications really don’t care about the routers inside their company. They know they are there, might know somewhat how they work, and as long as they work, its fine – that is what IT is all about. But on the other hand, the Network Administrator really cares about Cisco routers. He dreams about them. Names his kid Cisco, or Switch.

Now, the network admin can login to all his routers, and run some commands to backup his configs. The most usual way to do this is to send the config to a TFTP server. Now, if they want a backup once a month, and they have one router, well then great, a manual solution is fine. The network is probably not big or complex and the network admin needs something to do. In most cases though, they would want to back up their routers daily, and they might have multiple routers.

In this scenario, let the network admin set up the TFTP server. Those are abundant and easy to find, easy to setup. What we are concerned with from a development standpoint is actually logging into the router, running commands to backup the config (to the TFTP server) and getting out.

Now, a few things are needed from your network admin. First, you are going to need the IP addresses of all the routers. Next, you want to make sure that they have one user on all the routers with the same password that you can use just for this backup program. There are multiple ways I am sure they can do this, and since I am not a network guru, leave that to them – they will throw out terms like RADIUS, etc, but it should be easy for them. Next, you need them to make sure that all routers are set up the same, as far as the way they use “enable” commands, etc.

The first thing you want to do is take that information from your network admin, and then test each one manually. Telnet (or SSH if you can get that working) using the IP, login with the user and password, and then run the enable command, and look at the strings that are responsed back to you. Every router has a name like

company-router-123>

where the > is the prompt. You need to jot down this name to go along with the IP address. Now you can get fancy later and have your network admin set that name up in DNS and then you can just have a list of names, but start with IP addresses first.

Now, here comes the developing part. A long long time ago, right when .NET hit the airwaves, I created a class library called Winsock.Telnet so I could use it. Named it Winsock because I was a VB6 developer and I used the Winsock control to do telnets within my programs, so it just made sense. I still use this library today, and I do have the source code to it somewhere buried on a backup DVD or server in my apartment, and finding it would just be a wasted effort at this point, but the class library works, so that is what matters. I use this class library to do my telnets. (To do SSH I have used WeOnlyDo’s .net SSH Client – Chris Super blogs about how to run SSH on your network yet still use telnet for a specific purpose – such as this). You can get my Winsock library here.

Here is the guts of the main method to log a config from a Cisco router. Steps are easy. Connect, login, enable, run the TFTP command, send in the TFTP address, and a path , then exit. The second half is extra credit. I actually set up a SVN repo to the directory on the server that I TFTP the configs to, do a SVN diff, and if different, I email the changes to the network admin. But everything up the “exit” command would get you buy. The Sleep(1) function just waits for a second, which with telnet you need to do, so you don’t overrun your self. I have included the methods to do the SVN diff.

 

        private static void LogRouterConfigTelnet(string deviceName, string ipAddress, string enablePassword)
        {
            _connectorTelnet = new WinsockTelnet.Winsock(ipAddress, 23, 60);
            _connectorTelnet.Connect();
            _connectorTelnet.WaitAndSend("Username:", _username);
            _connectorTelnet.WaitAndSend("Password:", _password);
            _connectorTelnet.WaitAndSend(deviceName + ">", "enable");
            _connectorTelnet.WaitAndSend("Password:", enablePassword);

            Sleep(1);

            _connectorTelnet.SendAndWait("copy run tftp", "[]?");
            _connectorTelnet.SendAndWait(_tftpAddress, "?");
            _connectorTelnet.SendAndWait("routers/" + deviceName + "/" + _filename, deviceName + "#");
            _connectorTelnet.SendMessage("exit");
            _connectorTelnet.Disconnect();

            // copy over svn copies, delete from root folder
            File.Copy(@"C:TFTP-Rootrouters" + deviceName + @"" + _filename, @"c:tftp-sourcerouters" + deviceName + ".txt", true);

            // do svn diff
            string diff = SVNDiff(deviceName + ".txt");

            if (!string.IsNullOrEmpty(diff))
            {
                System.Console.WriteLine(diff);

                // if different, commit to svn, email diffs
                SVNCommit(deviceName + ".txt");

                EmailDiff(deviceName, diff.Replace(Environment.NewLine, "<br>").Replace("n", "<br>"));
            }


        }

        private static string SVNDiff(string filename)
        {
            ProcessStartInfo psi = new ProcessStartInfo();
            psi.FileName = @"C:Program FilesSubversionbinsvn.exe";
            psi.WorkingDirectory = @"C:tftp-sourceRouters";

            psi.Arguments = String.Format("diff {0}", filename);

            psi.UseShellExecute = false;
            psi.RedirectStandardOutput = true;
            psi.CreateNoWindow = true;

            Process p;
            String output;

            p = Process.Start(psi);

            try
            {
                output = p.StandardOutput.ReadToEnd();
                p.WaitForExit();

            }
            finally
            {
                // shouldnt happen but lets play it safe
                if (!p.HasExited)
                {
                    p.Kill();
                }
            }

            return output.Trim();

        }

        private static void SVNCommit(string filename)
        {
            ProcessStartInfo psi = new ProcessStartInfo();
            psi.FileName = @"C:Program FilesSubversionbinsvn.exe";
            psi.WorkingDirectory = @"C:tftp-sourceRouters";

            psi.Arguments = String.Format("commit -m "config changed" {0}", filename);

            psi.UseShellExecute = false;
            psi.RedirectStandardOutput = true;
            psi.CreateNoWindow = true;

            Process p;
            String output;

            p = Process.Start(psi);

            try
            {
                output = p.StandardOutput.ReadToEnd();
                p.WaitForExit();

            }
            finally
            {
                // shouldnt happen but lets play it safe
                if (!p.HasExited)
                {
                    p.Kill();
                }
            }

        }

        static void EmailDiff(string deviceName, string diff)
        {

            MailMessage msg = new MailMessage();
            msg.To = "networkadmin@yourcompany.com";

            msg.From = "ciscoconfig@yourcompany.com";
            msg.Subject = "Cisco Config Changed - " + deviceName;
            msg.Body = diff;
            msg.BodyFormat = MailFormat.Html;
            SmtpMail.SmtpServer = "yourmailserver";

            try
            {
                SmtpMail.Send(msg);
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine(ex.ToString());
            }

        }

        static void Sleep(int seconds)
        {
            System.Threading.Thread.Sleep(seconds * 1000);
        }

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }


 So, you can see, taking a little time to create a small program to do this is not really tough. And your IT department will be happy. It will also give you a reason to use things in .NET that you might not use everyday, especially if you are a Web Programmer, and also you will learn a little more about IT things (routers, networks, etc).

Note: you can see the code isn’t the prettiest, and really doesn’t need to be. There is some duplication yeah, and some hardcoded paths. If you are worried, release a 2.0 version with all that in the App.Config and refactor out a couple of methods. Or if you get really good, create a library called Utils or something with all the common functions you are going to use, like for calling processes, etc.

 

Data Recovery – Where Did My Files Go?

Right before I left for Portland, I was moving some files around on my USB hard drive, and I noticed something was wacked, like 100 GB was in use, but I couldn’t see any files! They had disappeared.

Well today I finally got around to recovering the files. I tried 3-4 different products at first. All of them let you download a demo and see what you can recover. The one I have used before – OnTrack – doesn’t even work on Vista, so that was out the window.

I finally came across one called GetDataBack. It showed EVERYTHING that I was missing. The others showed partials or not even some folders I had, just weird. GetDataBack is nice too, once you do a scan and it finds everything, you can save that recovery session, close the program, and come back, and you don’t have to rescan everything again. My scan took 3.5 hours to complete, so this was a nice feature.

The hardest part about everything is just moving data around, I didn’t have enough free space on my main drive to restore everything at once, so I had to restore some, and then move it over the network to my other machine, then go back and restore more, rinse, repeat.

GetDataBack set me back 70$, but it was worth it to get all my music, documents and photos back. I can tell you this, once I am done moving everything around, I am backing up everything online.

Vista – Hide Ad-Hoc Networks

Since I am getting my Internet today, this probably isn’t such a big deal, but for the past few days I have been “borrowing” the coffee shop’s wifi below my apt. Comes on in the morning, shuts off at night. But, there are like 20 wifi points in the list to connect to. I hate ad-hoc networks, they just seem shady.

So, if you want to hide them from the list:

netsh wlan add filter permission=denyall networktype=adhoc

To reverse this again run the following:

netsh wlan del filter permission=denyall networktype=adhoc