What's the recommendation about setting up indexes.conf which will be distributed (via deployment server) and also supports server specific shares? Basically, today all buckets are being stored on a LUN and I need to split off the colddb onto a CIFS share.
Here is the current config which all the Windows indexers have/receive:
splunk-launch.conf
SPLUNK_DB=E:\Splunk
indexes.conf
[default]
frozenTimePeriodInSecs = 126144000
lastChanceIndex = default
[volume:primary]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 7500000
[main]
homePath = volume:primary/defaultdb/db
coldPath = volume:primary/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
maxTotalDataSizeMB = 2000
After a bit of trial and error, I found out one cannot add additional variables to the splunk-launch.conf, but can use $COMPUTERNAME which Splunk will pull from the OS env variables. So this is now what I've been trying on one indexer:
indexes.conf
[default]
frozenTimePeriodInSecs = 126144000
lastChanceIndex = default
[volume:primary]
path = $SPLUNK_DB
maxVolumeDataSizeMB = 7500000
[volume:cold]
path = \\cifsdata.FQDN\SplunkColdData\$COMPUTERNAME
[main]
homePath = volume:primary/defaultdb/db
coldPath = volume:cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
maxTotalDataSizeMB = 2000
It appears to work. Took a few hours to migrate the few TB of colddb over, but I've ran into some new errors/oddities with this migrated indexer since then. Additionally, the splunk diag command doesn't resolve the $computername so it complains about every path.
Is there a better way to accomplish the same end goal?
↧
How to configure and distribute indexes.conf with server specific CIFS shares via deployment server?
↧
After changing data retention, why is indexes.conf not working on one indexer?
Hi
I have changed the data retention and pushed the bundle from the cluster master. In 2 indexers, the data got deleted but in one indexer, it's still the same. The indexes.conf in the slave-apps is same on all 3 indexers. What changes do I have to make so that the indexer will remove data from the cold buckets of one index?
Splunk version - 6.4.2
↧
↧
How to edit indexes.conf to resolve an error that says the index doesn't exist or is disabled when sending events?
I have several indexers checking into a deployment server and as such wanted to use a deployed app to manage indexes. In my environment, we have many indexes which results in using the deployment server to manage them a good solution.
The problem I'm having is that the indexes are getting created on the indexer, and it seems like it would be working just fine. However it's not, when sending events I get an error that says that the index doesn't exist or is disabled.
Here's the indexes.conf that the app pushes (right now it contains nothing else):
homePath = volume:hot1/index1/db
coldPath = volume:cold1/index1/colddb
thawedPath = $SPLUNK_DB/index1/thaweddb
The app is called:
SplunkIndexer-Linux/
I tried adding an export=system to the top of the apps indexes.conf and that didn't have an impact.
I've got to be missing something simple here... anyone have any ideas?
↧
What is the difference between using repFactor=0 versus repFactor=1?
Can anyone explain to me the differences between using repFactor=0 and repFactor=1?
We only replicate specific critical indexes in our environment (in an interest of conserving disk space). However, by not having repFactor set (so it defaults to 0), those indexes do not appear in the list of indexes in the indexer clustering UI and are not eligible for things like data rebalancing. So going back to my original question, would we get the ability to monitor the status and rebalance the data for these non replicated indexes if we set repFactor to 1 or is this just intended behavior of Splunk currently?
↧
Why are all my indexes disabled but Splunk is still writing data?
Hello colleagues,
Can you help me with the issue which I caught a couple days ago and I still couldn't resolve?
A couple days ago I tried to check my license status but I didn't do it because Splunk said that the data wasn't found.
When I tried to find result manually by doing a search request, I found that system indexes didn't have any events. After that, I checked settings and found that all indexes were disabled and I couldn't enabled through Splunk Web.
![alt text][1]
[1]: /storage/temp/174314-capture.png
I also checked splunkd.log and didn't find any Errors which might be related to my issue.
There is only this ERROR state `ERROR AuthenticationManagerLDAP - Could not find user="nobody" with strategy="mystrategy`
I did restart and passed all checks without any troubles.
I ran splunk btool check --debug to find something strange but didn't find anything.
After that, I had been observing folders for sometime which were used to internal indexes and detected that Splunk still was writing data.
I tried to enable an index by editing indexes.conf and putting to them disabled flag.
After restart Splunk showed me that the index had been enabled but there still wasn't any event there.
↧
↧
How to alter the amount of disk space used by Splunk?
Greetings;
I read some postings here that show how to adjust the sum-total disk space that SPLUNK uses. My root partition is below the magic 5GB limit, and my indexing has stopped. I've done this so far:
1) Looked inside /opt/splunk/etc/system/default/indexes.conf, and I saw the setting, "frozenTimePeriodInSecs" but the file says, "make no changes here, make changes in /opt/splunk/etc/system/local instead.
2) Files in this directory are inputs.conf, migration.conf and server.conf. I put, "frozenTimePeriodInSecs" in this file, and made it equal to 1.5 years (47,088,000 seconds)
Upon restarting SPLUNK, I'm now getting a message:
Checking conf files for problems...
Invalid key in stanza [default] in /opt/splunk/etc/system/local/inputs.conf, line 3: frozenTimePeriodInSecs (value: 47088000)
Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
Can anyone give me a rundown of the mathematical relationship between this value, and the other bucket-retention values, etc etc... so that my config is consistent?
↧
How do we read already indexed data when an index is deleted and re-created with the same name in an indexer cluster?
We have a clustered index with several indexers. While distributing Configuration Bundle from the cluster master, one of the index entries was taken out by mistake from indexes.conf file. After we put back the index entry in the indexes.conf file and redistributed the bundle, the index (recreated with the same name) cannot read the old data anymore, however, the old data still exists in the servers in the hot-warm buckets (ie ../db). I would really appreciate if someone can help point me to the right direction. We want to be able to read the old indexed data rather than re-indexing the data.
↧
Can I migrate indexes into a volume reference?
Hi,
Forgive me if this is a dumb question... can I move an index(s) into a volume reference?
Index paths will remain the same as we want to limit total size via "maxVolumeDataSizeMB".
Will the index name(s) be created in "coldToFrozenDir"?
I have an index(s) in indexes.conf (SPLUNK_DB=/media/data/splunk):
[sonicwall]
coldPath = $SPLUNK_DB/sonicwall/colddb
homePath = $SPLUNK_DB/sonicwall/db
thawedPath = $SPLUNK_DB/sonicwall/thawedd
[index n]
...
I wish for new indexes.conf to be:
coldToFrozenDir = /media/archive
[volume:san]
path = /media/data/splunk
maxVolumeDataSizeMB = 2000000
[sonicwall]
coldPath = volume:san/sonicwall/colddb
homePath = volume:san/sonicwall/db
thawedPath = $SPLUNK_DB/sonicwall/thaweddb
[index n]
...
Thank you.
↧
For clearing out data in an index with the wrong timestamp, will adjusting frozenTimePeriodInSecs in indexes.conf work in this case?
Hello guys,
I had wrongly indexed data with bad timestamp and wanted to clear them in an index and tried frozenTimePeriodInSecs=86400.
It didn't work, is it normal? Only the `| delete` works in this case?
Many thanks.
↧
↧
Why has data archiving stopped after archiving only two buckets?
Hi,
I have data more than 2 years old in Splunk. I want to archive data which is older than 18 months to AWS s3 buckets.
For this I add following settings in indexes.conf:
frozenTimePeriodInSecs = epoch for 6 months
coldToFrozenDir = path to the s3 buckets
But still archiving has been stopped after archiving two buckets. There is no error in _internal logs.
Could anybody please help in this?
↧
My index size limit has been reached. Why is oldest indexed data not being deleted "in order"?
I have an index with a size limit of 80GB and based on the data we index this should be about 7-10 days of retention.
However I have been indexing into that index for 3 months and it's full and old data being deleted as expected.
But it's messy! Rather than having a clean 7-10 days of data with a clear line where the old data is being deleted I have some entries span the whole 3 months.
**1-10 days old** = 50m entries per day
**11-25 days old** = 100k entries per day
**26-70 days old** = 10k entries per day
**70-90 days old** = 0-100 entries per day
No other configurations change. Data comes in in chronological order.
Anyway to make this clearer so that analyst don't have this incomplete data span a huge time range. Just a complete data set for 7-10 days.
I'm sure this is an obvious one but couldn't find anything so far in manuals/forum.
Cheers...
↧
How to troubleshoot why hot buckets are not rolling after exceeding maxHotSpanSecs?
Hi All,
My hot bucket is not rolling when its span has exceeded maxhotspansecs. Could you please provide assistance?
We are currently using a Splunk index, purely for data archiving purposes with the requirements as per below:
- The data will be captured in single bucket of 24hour period for Reingestion purposes.
- The hot bucket will roll straight from Hot to Cold.
- Data will sit in cold for 6 days
- Data will roll to frozen after a period of 7 days.
After applying the configuration (indexes.conf as per below): I have noticed that the bucket span has exceeded 86401 as defined.
Bucket Start epoch time: 1481822441
Bucket End Epoch time: 1482106850
Hence Span sec = 284409 - which is greater than 86401.
Indexes.conf Snippet:
[my_index]
frozenTimePeriodInSecs = 604800
maxTotalDataSizeMB = 400000
maxWarmDBCount = 0
maxHotSpanSecs = 86401
maxHotBuckets = 1
coldToFrozenDir =
Kind regards,
Craig
↧
Hunk: Why am I unable to retrieve results for a date and time range beyond today's date?
I have the exact same issue as https://answers.splunk.com/answers/320535/post.html .
I tried the regex provided in the solution but it still doesn't work.
To elaborate,
We have data in directories in the following format
/.../../buildlogs/time_slot=201701070930/......
for data of 2017 Jan 7th 9:30 AM.
Following is my indexes.conf excerpt
vix.provider = abc
vix.input.1.splitter.hive.tablename = logger
vix.input.1.splitter.hive.fileformat = orc
vix.input.1.splitter.hive.dbname = logsdb
vix.input.1.path = hdfs://xyz/buildlogs/...
vix.input.1.splitter.hive.columnnames = contents,project
vix.input.1.splitter.hive.columntypes = string:string
vix.input.1.required.fields = timestamp
vix.input.1.ignore = (.+_SUCCESS|.+_temporary.*|.+_DYN0.+)
vix.input.1.accept = .+$
vix.input.1.et.format = yyyyMMddHHmm
vix.input.1.et.regex = .*?/time_slot=(\d+)/.*
vix.input.1.lt.format = yyyyMMddHHmm
vix.input.1.lt.regex = .*?/time_slot=(\d+)/.*
vix.input.1.et.timezone = GMT
vix.input.1.lt.timezone = GMT
vix.input.1.lt.offset = 1800
vix.input.1.et.offset = 0
If i search with a date range of any day other than today , I get no results. ![alt text][1]
However if i search with the date range of today , I get valid results. ![alt text][2]
I verified that data from previous days is present in Hive.
Can someone help?
[1]: /storage/temp/177178-screen-shot-2017-01-09-at-125823-pm.png
[2]: /storage/temp/177179-screen-shot-2017-01-09-at-10010-pm.png
↧
↧
Currently audit_db, access_summarydb are consuming space in PCI Search Head. How do I find the indexes.conf file and its configurations?
Hi All, Currently we have noticed under this path /opt/splunk/var/lib/splunk/ audit_db & access_summarydb has occupied more space in PCI search head. I am trying to find out what is the retention period configured in the indexes.conf but I am unable to find the indexes.conf file in the PCI search head.
How to find the exact configuration using the Splunk cmd btool command? Though I had tried below command it listed out entire list in that instance .
./splunk cmd btool indexes list --debug
Distributed Environment and we are using Splunk version 6.2.1 with 4 search head, 5 indexer, 1 search job and license /deployment manager.
Kindly guide me in fixing this issue.
thanks in advance.
↧
Splunk Analytics for Hadoop: Why is Hunk searching all of the HDFS files instead of restricting it to the selected the time range?
We are new to Hunk (or now called Splunk Analytics for Hadoop).
I am attempting to run a query on our HDFS directories for the last 5 mins.
Here is the query: `index=foo | sort 0 _time`
So just return all the entries from the last 5 mins in the index foo sorted without truncation.
But it searches through all 8 million + events in our HDFS directories even after it seems to have found the complete list for the last 5 mins.
Any reasons why it might be doing this?
↧
Splunk Analytics for Hadoop: Why is Splunk not reading current active HDFS file?
We are running Splunk Analytics for Hadoop v6.5.1 with Hortonworks HDP v2.5.
I can search and results are returned within the timerange **EXCEPT** for the current file. There are no results returned if I am searching for events in the current hour. I'm not sure what the difference is. **Can someone help me troubleshoot?**
----------
The file is being written to using webhdfs:
[http://docs.fluentd.org/articles/out_webhdfs][1]
There is a new file created on the hour, hdfs structure is as follows:
/syslogs/yyyy/yyyy-MM-dd_HH_datacollectorhostname.txt
eg.
/syslogs/2017/2017-01-11_16_datacollector2.txt
Here is some sample data:
hdfs@mynn1:~$ hadoop dfs -tail /syslogs/2017/2017-01-11_16_datacollector2.txt
2017-01-11T21:59:59Z syslog.tcp {"message":"<167>2017-01-11T21:59:59.976Z myhost.internal Vpxa: verbose vpxa[259FBB70] [Originator@6876 sub=hostdstats] Set internal stats for VM: 878 (vpxa VM id), 314 (vpxd VM id). Is FT primary? false","client_host":"10.0.0.30"}
Here are the contents of my indexes.conf:
[provider:myprovider]
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-hy2.jar
vix.env.HADOOP_HOME = /usr/hdp/2.5.0.0-1245/hadoop
vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.10.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.6.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-exec-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-metastore-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-serde-1.2.1.jar
vix.env.JAVA_HOME = /usr/lib/jvm/java-8-oracle
vix.family = hadoop vix.fs.default.name = hdfs://mynn1.internal:8020
vix.mapred.child.java.opts = -server -Xmx1024m -XX:ParallelGCThreads=4 -XX:+UseParallelGC -XX:+DisplayVMOutputToStderr
vix.mapreduce.framework.name = yarn
vix.output.buckets.max.network.bandwidth = 0 vix.splunk.home.hdfs = /user/splunk/splunk-srch/
vix.yarn.resourcemanager.address = mynn2.internal:8050
vix.yarn.resourcemanager.scheduler.address = mynn2.internal:8030
[hdp-syslog]
vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.regex = /syslogs/(\d+)/\d+-(\d+)-(\d+)_(\d+)_\w+\.txt
vix.input.1.et.offset = 3600
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.regex = /syslogs/(\d+)/\d+-(\d+)-(\d+)_(\d+)_\w+\.txt
vix.input.1.lt.offset = 3600
vix.input.1.path = /syslogs/...
vix.provider = myprovider
Here is the contents of my props.conf
[source::/syslogs/...]
sourcetype = hadoop
priority = 100
ANNOTATE_PUNCT = false
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
TZ=UTC
[1]: http://docs.fluentd.org/articles/out_webhdfs
↧
What is the difference between putting configuration files under the Splunk Search app and other apps?
May I know the difference between putting configuration files under search-app or any other newly created app/slave-apps in indexers? If we put config files (indexes.conf and props.conf) under search-apps, can we access the index data through other apps?
↧
↧
How to resolve the indexer file system from reaching the threshold on /splogs?
Our's is distributed environment with 2 search heads , 5 indexers .
one of the indexers /splogs is 86% filled up , we set a threshold of 88% .
In /opt/splunk/etc/apps/HSY-ADMIN-all_indexers/local , i can see the configuration is set as
indexes.conf file is set as below
[volume:hsyHot]
path = /splogs
[volume:hsyCold]
path = /splogs
[volume:hsyBase]
path = /splogs
[default]
frozenTimePeriodInSecs = 31536000
Kindly suggest me what can i do reduce the size , what needs to be added to control the size on that .
Thanks
↧
If maxDataSize in indexes.conf is for hot buckets, then what parameter needs to be defined to set the size for warm and cold buckets?
In indexes.conf, it is given that "maxDataSize: The maximum size in MB for a hot DB to reach before a roll to warm is triggered". What parameter is defined to set size for warm buckets and the cold buckets?
↧
Index Size : Calculate maxTotalDataSizeMB
Hello Team,
I have some confusion on calculating maxTotalDataSizeMB for configuring in indexes.conf file. Below are the details:
Daily Data volume: 400GB
Retention Period: 90 days
Number of indexers in cluster: 20
Search Factor: 2
Replication Factor: 3
What will be the value of maxTotalDataSizeMB parameter in indexes.conf file for a particular index. Will it be (400*90*1024)MB or 400*90*1024 divide by 20 indexers. If maxTotalDataSizeMB is low then data will be deleted before retention. What is the optimum size for this?
[index]
homePath = volume:primary/index/db
coldPath = volume:primary/index/colddb
thawedPath = $SPLUNK_DB/index/thaweddb
tstatsHomePath = volume:primary/index/datamodel_summary
maxTotalDataSizeMB = 36864000????
frozenTimePeriodInSecs = 7776000
Thanks
Hemendra
↧