43

I've been handed 3 Linux boxes, 1 front facing with apache on it and another 2 which, as far as I can tell, don't do an awful lot. All running on Redhat.

The question is simple: How can I tell what the server is actually doing? Zero documentation is available from the creator.

Aaron Hall
  • 296
  • 3
  • 12
Bizmark
  • 598
  • 4
  • 5
  • 7
    The processlist, network listeners (maybe compare to would should have been running based on init scripts) ... – HBruijn Jun 09 '15 at 13:27
  • "handed" as in given? (for your own use?) - If so, wipe and fresh install, use for what you want, does it matter what they did? – Digital Lightcraft Jun 09 '15 at 13:33
  • 1
    "handed" as in given to support :| – Bizmark Jun 09 '15 at 13:47
  • 5
    damn! surely someone knows what they are for? so they want you to support them but dont know what they do?! – Digital Lightcraft Jun 09 '15 at 14:25
  • 2
    Incredibly, this does happen. I've been in these shoes. – Aaron Copley Jun 09 '15 at 14:55
  • 3
    It definitely happens. I'm _two weeks_ into the discovery process in an orphaned undocumented environment. But the difference is that I know what to look for... To the OP, how did you end up in this situation? We can answer in general terms, but there has to be someone who knows more about the setup. – ewwhite Jun 09 '15 at 15:03
  • The front-facing one might be a load balancer which directs traffic to the other two based on their load. – MonkeyZeus Jun 09 '15 at 16:58
  • 11
    Turn them off. Someone will let you know, *almost immediately*, what's not working. – jscott Jun 09 '15 at 17:15
  • 3
    As jscott said - also, in switched-off state they still perform all documented tasks :) – Hagen von Eitzen Jun 09 '15 at 17:44
  • 58
    **DO NOT TURN IT OFF.** Unplug the Ethernet cable, if you want to scream-test. If you've never had a box with a two-year uptime fail to reboot, you will at some point. This isn't the time to add that frustration to the mix. – Aaron Copley Jun 09 '15 at 18:16
  • 1
    @AaronCopley Very wise words. I have been there myself. And for the record: I didn't turn them off. I received 8 of the things in a shipping crate. Several failed disks, 3 power-supplies. And 2 couldn't fully recover a dirty raid: They had apparently just yanked the power-cords and the backup-batteries on those raid-controller caches was long dead. – Tonny Jun 09 '15 at 18:49
  • 7
    What everyone else said, but also: run nmap against them. – Katherine Villyard Jun 09 '15 at 21:11
  • To answer the question of how I got this: it's a new support contract that's been handed over. Previous contract was less formal and required very little documentation. Some of the stuff goes back 5 years, so even if they old company wanted to help us the technicians who implemented the servers are long gone. – Bizmark Jun 10 '15 at 13:28
  • 2
    Use a local packet sniffer (e.g. `wireshark`) to watch how the three interact with each other and the world. You can record for a while, filter the recording various ways, corroborate port numbers etc. -- **the whole picture will be there** if you know how to look at it. – goldilocks Jun 10 '15 at 16:27
  • [Related question about a Windows 2003 server](http://serverfault.com/questions/661000/how-to-identify-who-what-uses-a-windows-2003-server/661005#comment805974_661005). Top-voted answers are not OS-specific. – Lilienthal Jun 11 '15 at 11:14
  • @goldilocks suggestion is good - if you can't tcpdump/Wireshark directly, maybe you can mirror the switch ports for the servers and log the traffic on a workstation without interfering with it. – TessellatingHeckler Jun 11 '15 at 21:09
  • I second what Katherine said about running nmap against it (from a remote machine). It is useful to give you an "inventory" of sorts of any open ports that could potentially be in active use (thus providing helpful hints as to its function), but are not necessarily always significant. In some cases you can even telnet to different ports just to see if you get a response or if they are listening to incoming connections. Equally useful if you have access to the server directly use netstat -an – SeligkeitIstInGott Jun 12 '15 at 15:56
  • While this question is extremely and overly broad - it isn't at the same time. Many of us have been faced with this same challenge, and even though our specific solution will vary greatly, the fact of the matter is, many new professionals simply don't know where to even start. This question seems to propose a good starting point for anybody, even highly experienced professionals to start. A checklist would be great! – IceMage Jun 12 '15 at 20:27

6 Answers6

43

Unplug the ethernet cable and see who gets upset.

Seriously though, mystery machines like this create a lot of mental overhead for a team and often provide absolutely no business value. Talk to your boss, if no one knows what it does maybe no one cares what it does.

Josh Rumbut
  • 573
  • 5
  • 11
  • 44
    **DO NOT TURN IT OFF.** Unplug the Ethernet cable, if you want to scream-test. If you've never had a box with a two-year uptime fail to reboot, you will at some point. This isn't the time to add that frustration to the mix. (Copying this here because it needs to be read.) – Aaron Copley Jun 09 '15 at 18:17
  • 3
    You're 100% right, I've edited the post to reflect that. I was being a little facetious but I can still be facetious without promoting disaster. – Josh Rumbut Jun 09 '15 at 18:44
  • 8
    Before the network is unplugged it is a good idea to save a list of running processes and open sockets on each server - just in case something is relying on a TCP connection between the servers which happens to have remained ESTABLISHED for many months without anybody thinking about automating whatever steps were needed to open it in the first case. (For example if somebody needed an ssh port forwarding temporarily and then forgot about it.) – kasperd Jun 09 '15 at 21:52
  • 4
    Don't do this. What ridiculous advice. Stupid thoughtless IT people doing this cost me so much time and work. ASK FIRST. If you don't know who to ask, _ASK EVERYONE_. – Lightness Races in Orbit Jun 10 '15 at 20:16
  • 4
    Make sure you leave sufficient time for someone to scream - the one time I did this with a dusty or desktop machine sitting on a server room rack - no one knew what it did, it took a full month for someone to notice when we unplugged it. Turns out it was part of the payroll system, without that server, Accounting couldn't generate monthly payroll. It had been running unattended for at least 3 years with no one paying any attention to it -- so the "scream test" was a success, if we hadn't done it, the server would have eventually died on its own. We ended up p2v'ing it into our vmware cluster. – Johnny Jun 10 '15 at 23:32
  • 2
    Yeah, turning it off, while amusing from a visceral standpoint, probably isn't smart and [may get you in a lot of hot water](http://arstechnica.com/information-technology/2014/10/it-came-from-the-server-room-halloween-tales-of-tech-terror/?comments=1&post=27873759) – Machavity Jun 11 '15 at 13:03
31

This is a pretty broad question for the Serverfault format, but here is a good start:

  • Check for running processes and those scheduled to run at system startup.
    • Review the running configuration of each.
    • Look into any defined data directories. (Maybe someone installed MySQL and turned it on, but there are no databases.)
  • Check for scheduled tasks.
  • Check the logs to see;
    • who has logged in recently (and ask them)
    • and to get an idea of what's been running.

You didn't mention the version, so I've omitted the specifics.

Aaron Copley
  • 12,525
  • 5
  • 47
  • 68
  • 8
    There is something more important than which services are configured to start when the system boots. Which services are running right now? Starting a service and forgetting to configure it to start at boot is not a difficult mistake to make. On a related note, it is a good idea to look at other system state such as: mount points, routing table, iptables rules. All of those are things which could easily be changed while the system is running without remembering to update the config files used during boot. – kasperd Jun 09 '15 at 21:58
  • In addition I would use a port scanner to see which ports are open, then try to connect to it using the usual tools. For (a simple) example if port 443 is open you could try using a web browser to connect to it. I have had to explore such seemingly abandoned servers frequently and one of my favourite tools to quickly browse through the configuration files in /etc and other places is to use "lynx" or "links" if you prefer. These are character based web browsers which do a good enough job as a file browser as well, with convenient cursor key navigation. – aseq Jun 09 '15 at 22:43
  • 1
    @kasperd Fair play on running vs. persistent services. But, I thought about firewall rules, mount points, etc. Those seemed to me as ancillary components that will already be tied to one of the existing bullet points. YMMV. – Aaron Copley Jun 09 '15 at 23:18
  • Please add - Check for active network connections, and write down the names of the services and port numbers. The best method to do this varies by operating system. E.G. net stat. Also, put some kind of trace on the computer so you can see what it does throughout the day. You may also want to add consideration for scenarios when things running on the servers COULD be malicious. – IceMage Jun 12 '15 at 20:28
19

There are a few things you could do to try and ascertain what's running on your system.

You can check which ports your server is listening on to get an idea of what's on there. A good command to use would be:

 [root@server ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             Stat    e       PID/Program name
tcp        0      0 0.0.0.0:139                 0.0.0.0:*                   LIST    EN      1880/smbd
tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LIST    EN      1911/nrpe
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LIST    EN      1759/sshd

As you can see from the example output above, it presents you with the protocol version (tcp or udp), the address that's being listened on, the port that's open and the program that's listening.

In the above truncated example (a server machine) you can see tcp ports 139, 5666, and 22 are listening. These resolve to samba, nrpe (Nagios agent) and ssh respectively, and is confirmed when you check the program that's listening on that port.

Additionally, you can check the list of daemons which are configured to start at boot, in order to do that, run: chkconfig --list | grep "3:on"

Example:

[root@server ~]# chkconfig --list | grep "3:on"
NetworkManager  0:off   1:off   2:on    3:on    4:on    5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sysstat         0:off   1:on    2:on    3:on    4:on    5:on    6:off
udev-post       0:off   1:on    2:on    3:on    4:on    5:on    6:off
vncserver       0:off   1:off   2:on    3:on    4:on    5:on    6:off
webmin          0:off   1:off   2:on    3:on    4:off   5:on    6:off
x2gocleansessions       0:off   1:off   2:on    3:on    4:on    5:on    6:off
.
.
.

or :

service --status-all

Itai Ganot
  • 10,644
  • 29
  • 93
  • 146
18

Another method involves checking the /etc directory and looking at the modification dates. After a fresh install all the files in this directory should have roughly the same date/time. And since an install usually installs a lot of things people usually do not use, only the files that have a later modification date reflect the actual purpose of the server. If this is ext4 you also should be able to extract the birth date of directories, so the task could be quite easy.

Yet another method would involve checking the .bash_history files to see what the admins were up to. This file can provide a wealth of knowledge.

sebix
  • 4,313
  • 2
  • 29
  • 47
Konrad Gajewski
  • 1,518
  • 3
  • 15
  • 29
7

Check the firewall rules. With a bit of luck, it's configured for default-deny. That means there's an explicit rule for each allowed service.

This is better then netstat because it can also show ports that are open for e.g for nightly backups.

MSalters
  • 700
  • 5
  • 6
6

One answer I've not seen yet: Check the most recently modified files. Logs, database files, other output files etc. may get written to still that may provide clues:

find . -mtime -3

That would find modified files in the current directory and deeper, changed in the last 3 days. Increase the number 3 to an educated guess until you get some output you can investigate.

Not fool-proof, as the boxes may just process some web service calls, returning some data without ever writing anything. But added to the great mix mentioned above, it may just yield some clues.

JayMcTee
  • 3,923
  • 1
  • 13
  • 22