half ass junkies

Some are proud to be sysadmins, but think that they are doing a fine job. This idiot set up a splunk forwarder on a host and ran it as non-root(splunk). After a few weeks, he complains that the indexer is not receiving any data from this forwarder. Since I am a contractor, I need to do the dirty job.

When I run splunkd as root, indexer receives data (kprocess <> 0); however, when it is run as non-root (that is, by invoking /etc/init.d/splunk {start|stop|restart}), indexer doesn't receive any data from that forwarder (kprocessed = 0).


3.2 index=_internal source="fwd-hb" host="x.y.z.net" kprocessed=0
4.0 | metadata type=hosts | eval age = now() - lastTime | search age > 86400 | sort age d | convert ctime(lastTime) | fields age,host,lastTime
4.0 | metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time


chown -R splunk $SPLUNK_HOME fixed the mess. It took a damn day. This idiot set up all splunk clients at $WORK, yet doesn't follow the documentation on how to run splunk as non-root user. If you don't follow the documentation, that's fine, assuming that you know how splunk works. The problem is: you don't know how it works, you don't follow instructions, and you frigging waste others time, and whine!!