Down the rabbit hole – Splunk Add-on for Microsoft Cloud Services key rotation

When you need to gather data from an Azure Storage account like the Exported Application Insights data. Then you need an addon for it, such as the addon “Splunk Add-on for Micorosoft Cloud Services” that is frequently used. The addon works ok you set up an Azure Storage Account Configuration, configures an input channel to the index you want with an interval, and besides that, the addon doesn’t clean its own old data (volume full exceptions in the splunkd log) everything goes great.

That is until you need to rotate the storage account keys. Just go into the Azure account configuration, press edit and paste in the new key. Got a nice message that key was successfully updated and no errors so good, right?

Validated the changes and it seemed data was coming in, still don’t know how that happened. After an hour I noticed that the egress on my Azure storage account was way lower than expected and that the data was not flowing to Splunk. Maybe there was an additional character copied during the key rotation. So did the action again, with the same result. Next, a whole lot of investigation happened but none of the logs gave any insights into the problem nor gave any error.

index=_internal log_level=ERROR mscs:storage:blob:log

Because I wanted to understand the problem I went into investigation mode. To my surprise, the solution was more obvious than expected.

Somehow the TA didn’t restart itself and you needed to do a manual disable/enable.

I reported the issue to Splunk, together with the info that the TA doesn’t clean up the local directory after indexing. If the log gave me some kind of hint it would be solved way faster, finding solutions on the internet didn’t help either.

I hope that this little fun article saves somebody some time.

Erick