Resolving Elasticsearch Data Acceptance Issues Due to Disk Watermark Limit

Last update: 2026-02-17
Last verified version: AAC 4.4.3

Overview

This article explains how to resolve issues when Elasticsearch stops accepting data because the disk watermark limit is reached.

Elasticsearch uses a disk watermark to decide when to enter read-only mode. If the disk is nearly full, it stops accepting new data. The default watermark is 95%, preventing data loss by halting writes when space is low.

The Analysis Cockpit sets its disk watermark to 90%. On a 1TB drive, you need at least 100GB free to avoid read-only mode. This lower threshold gives you more time to act before Elasticsearch requires a manual reset to exit read-only mode.

Prerequisites

The below message shows up in the Analysis Cockpit if the disk watermark is reached:

If you see this message, the Analysis Cockpit went into a read-only mode and you need to free up some disk space or increase the disk space by allocating more storage to the virtual machine.

Expected result

Elasticsearch accepts data again. The Cockpit is working properly

Steps to reproduce

Step 1: Freed up Disk Space

Freed up disk space by following these articles:

• ASGARD Analysis Cockpit Malfunction Due to Full Disk: Solutions and Steps to Restore Functionality

• Extending ASGARD's Disk Space

• Managing Disk Space in Analysis Cockpit: Safe and Unneeded File Deletion Options

Step 2: Reset Elasticsearch State

If your disk usage somehow got above 95%, Elasticsearch will go into read-only mode nonetheless. If this happened, you need to reset the Elasticsearch state after you freed up some disk space. You can achieve this by running the following command:

nextron@cockpit:~$ curl -X PUT -s -u elastic:$(cat /etc/asgard-analysis-cockpit/elastic.password) \
-H 'Content-Type: application/json' \
-d '{"index.blocks.read_only_allow_delete": null}' \
http://localhost:9200/_all/_settings

You should get the following output if the command was successful:

{"acknowledged":true}